< Niklaus.20210126-ruby-30 | Index | Niklaus.20200615-migel-utf-8-ruby-271 >>
YDBI:
YDOCX:
htmlgrid:
odba:
oddb.org:
Installing Ruby 3.0.0 using /usr/local/src/gen_ruby_300 Attach:gen_ruby_300
Updated the soft links in /usr/local/bin for ruby and bundle to point to ruby_300, resp. bundle_300.
Running bundle update provoked errors like this when compiling syck
emitter.c: At top level: cc1: warning: unrecognized command line option ‘-Wno-self-assign’ cc1: warning: unrecognized command line option ‘-Wno-parentheses-equality’ cc1: warning: unrecognized command line option ‘-Wno-constant-logical-operand’ cc1: warning: unrecognized command line option ‘-Wno-cast-function-type’
After disabling all GCC warnings via bundle config build.syck --with-cflags="-w" I got the following error
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
current directory: /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0/ext/syck
/usr/local/ruby-3.0.0/bin/ruby-300 -I /usr/local/ruby-3.0.0/lib/ruby/site_ruby/3.0.0 -r ./siteconf20210105-22829-4lhsnc.rb extconf.rb --with-cflags\=-w
checking for st.h... no
creating Makefile
current directory: /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0/ext/syck
make DESTDIR\= clean
current directory: /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0/ext/syck
make DESTDIR\=
compiling bytecode.c
compiling emitter.c
compiling gram.c
compiling handler.c
compiling implicit.c
compiling node.c
compiling rubyext.c
rubyext.c:26:20: error: field ‘klass’ has incomplete type
struct RClass klass;
^[[~]] January 05, 2021, at 06:09 PM
Makefile:243: recipe for target 'rubyext.o' failed
make: *** [rubyext.o] Error 1
make failed, exit code 2
Gem files will remain installed in /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0 for inspection.
Results logged to /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/extensions/x86_64-linux/3.0.0-static/syck-1.4.0/gem_make.out
An error occurred while installing syck (1.4.0), and Bundler cannot continue.
Make sure that `gem install syck -v '1.4.0' --source 'https://rubygems.org/'` succeeds before bundling.
In Gemfile:
syck
Observationss:
Now many tests run, but parsing fachinfo/patinfo have errors.
Decided with Zeno to remove ydim and sych. Committed changes locally on oddb-ci2.
Running sudo -u apache bundle-300 exec ruby-300 jobs/import_daily to test the changes.
Parsing using hpricot is different with ruby 3.0. Parsing the HTML show that somehow <p> are no longer correctly transformed into paragraphs. Adding a very simple test case to try to fix this problem. Extracted TestPatinfoHpricot into ext/fiparse/test/test_hpricot.rb and added some more tests for paragraphs and images, as I get errors like
ODDB::FiParse::TestPatinfoHpricot#test_chapter_ponstan_with_sections: NoMethodError: undefined method `next_paragraph' for "Für Erwachsene und Jugendliche über 14 Jahre\n ":String /opt/src/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:196:in `handle_element'
Updated /service/migeld/run to use exec sudo -u apache ruby-300 bin/migeld by updating Gemfile and Gemfile.lock.
Calling sudo gem-300 install ycurrency; sudo gem-300 install rclconf, also calling sudo -u apache /usr/local/bin/ruby-300 /usr/local/ruby-3.0.0/bin/currencyd in /service/currency/run
Callingsudo gem-300 install yus. Updating /service/yus/run to call sudo -u apache /usr/local/bin/ruby-300 /usr/local/ruby-3.0.0/bin/yusd
Also updated /service/ch.oddb-meddata/run and /service/ch.oddb-refdata/run to use ruby-300 instead of ruby-272.
Discovered that running import_daily generates a OutOfMemory error. See https://github.com/zdavatz/oddb.org/issues/123.
Running sudo -u apache bundle-300 exec ruby-300 jobs/update_textinfo_swissmedicinfo pi to see, whether all patinfo can be parsed.
It looks like ODBA does not handle correctly empty arrays. Must update ydbi and odba for ruby 3.0, as there are various errors when running the tests.
Also ydocx tests have errors when running under Ruby 3.0. ydocs also should require 'rmagick' and not the out-dated 'RMagick', as gem name should always be lowercase.
It looks to me, that there are invalid fachinfos in our database, as on thinpower I get the following output
ch.oddb> fachinfos.find_all{ |x,y| y.iksnrs.size == 0 && y.localized_name == nil}.size
-> 239
Modifying in src/view/fachinfo.rb the method to_html to skip these invalid entries. Now update_textinfo_swissmedicinfo pi finishes, but memory useage is still very high (95% of my 16GB RAM, thinpower has 32GB). Adding some sleeps to allow odba cache cleanup to work. Also adding some debug info, to see how much memory is used to pinpoint the problem. Now the memory usage while updating the RSS stays at less than 3,5 GB, but prolongs the time needed to finish by around 400 seconds = 7 minutes.
Checking whether I can reduce this a little bit. Done. Running import_daily, then import_bsv_followers.
After many cleanups in htmlgrid, spec and unit tests the unit tests now pass all under Ruby 3.0. Under ruby head they fail, because there is not yet a nokogiri version. Under ruby 2.7 there are two obscure errors related to odba (I think).
There are stil some rspec failures. I do not want removed, because I am quite confident, that the expected behaviour is correct, but the test cases take at the moment too much time to fix them. I will document the failing tests here:
1) ch.oddb.org desitin should find redirect an iphone to the mobile flavor
Failure/Error: expect(iphone_browser.url).to match(/\/\/i\./) # this fails
2) ch.oddb.org desitin should display the correct calculation for Bicalutamid Sandoz
Failure/Error: tageskosten = @browser.trs.find{|x| /^Tageskosten/.match(x.text)}.text
3) ch.oddb.org desitin should have a working instant search for Inderal and going back
Failure/Error: expect(inhalt).to match(/Preisvergleich f??r/i)
4) ch.oddb.org desitin limiting the search to search_limitation_E using Holunder
Failure/Error: expect(nr_unrestricted_first).to be > nr_restriced if must_be_greater
<..>
1) ch.oddb.org should have a working RSS-feed /de/gcc/rss/channel/fachinfo
Failure/Error: @browser.goto(link.href) if link
Failure/Error: @browser.goto OddbUrl
<..>
1) ch.oddb.org check pharmacy
Failure/Error: @browser.link(visible_text: Fridolin).click
1) ch.oddb.org should be possible to run grant_download oddb2.csv
<..>
Full Log is here: Attach:spec_2021_01_22_log.txt
But how should I fix the ATC-Importer. Last time I changed something substantial was were the following commits