view · edit · sidebar · attach · print · history

< Niklaus.20210126-ruby-30 | Index | Niklaus.20200615-migel-utf-8-ruby-271 >>

20210105-ruby-30

Summary

Commits

YDBI:

YDOCX:

htmlgrid:

odba:

oddb.org:

Index

Port oddb.org to Ruby 3.0

Installing Ruby 3.0.0 using /usr/local/src/gen_ruby_300 Attach:gen_ruby_300

Updated the soft links in /usr/local/bin for ruby and bundle to point to ruby_300, resp. bundle_300.

Running bundle update provoked errors like this when compiling syck

emitter.c: At top level:
cc1: warning: unrecognized command line option ‘-Wno-self-assign’
cc1: warning: unrecognized command line option ‘-Wno-parentheses-equality’
cc1: warning: unrecognized command line option ‘-Wno-constant-logical-operand’
cc1: warning: unrecognized command line option ‘-Wno-cast-function-type’

After disabling all GCC warnings via bundle config build.syck --with-cflags="-w" I got the following error

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

current directory: /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0/ext/syck
/usr/local/ruby-3.0.0/bin/ruby-300 -I /usr/local/ruby-3.0.0/lib/ruby/site_ruby/3.0.0 -r ./siteconf20210105-22829-4lhsnc.rb extconf.rb --with-cflags\=-w
checking for st.h... no
creating Makefile

current directory: /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0/ext/syck
make DESTDIR\= clean

current directory: /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0/ext/syck
make DESTDIR\=
compiling bytecode.c
compiling emitter.c
compiling gram.c
compiling handler.c
compiling implicit.c
compiling node.c
compiling rubyext.c
rubyext.c:26:20: error: field ‘klass’ has incomplete type
struct RClass  klass;
^[[~]] January 05, 2021, at 06:09 PM
Makefile:243: recipe for target 'rubyext.o' failed
make: *** [rubyext.o] Error 1

make failed, exit code 2

Gem files will remain installed in /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/gems/syck-1.4.0 for inspection.
Results logged to /home/niklaus/git/oddb.org/vendor/ruby/3.0.0/extensions/x86_64-linux/3.0.0-static/syck-1.4.0/gem_make.out

An error occurred while installing syck (1.4.0), and Bundler cannot continue.
Make sure that `gem install syck -v '1.4.0' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
     syck

Observationss:

  • I was able to compile locally the syck gem (and use it for oddb.org)
  • ydim needed a small patch to pass the test
  • oddb.org uses also the deprecated https://github.com/ruby/mathn. Here I think it would be better to use the new type system of ruby 3.0
  • Running test/test_plugin/text_info.rb (test/test_plugin/text_info.rb lead to
    • mechanize.rb:12:in `require': cannot load such file -- webrick/httputils (LoadError)
  • Must add new (no longer stdlib) gem webrick and rss to the Gemfile

Now many tests run, but parsing fachinfo/patinfo have errors.

Decided with Zeno to remove ydim and sych. Committed changes locally on oddb-ci2.

Running sudo -u apache bundle-300 exec ruby-300 jobs/import_daily to test the changes.

Parsing using hpricot is different with ruby 3.0. Parsing the HTML show that somehow <p> are no longer correctly transformed into paragraphs. Adding a very simple test case to try to fix this problem. Extracted TestPatinfoHpricot into ext/fiparse/test/test_hpricot.rb and added some more tests for paragraphs and images, as I get errors like

ODDB::FiParse::TestPatinfoHpricot#test_chapter_ponstan_with_sections:
NoMethodError: undefined method `next_paragraph' for "Für Erwachsene und Jugendliche über 14 Jahre\n ":String
/opt/src/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:196:in `handle_element'

Updated /service/migeld/run to use exec sudo -u apache ruby-300 bin/migeld by updating Gemfile and Gemfile.lock.

Calling sudo gem-300 install ycurrency; sudo gem-300 install rclconf, also calling sudo -u apache /usr/local/bin/ruby-300 /usr/local/ruby-3.0.0/bin/currencyd in /service/currency/run

Callingsudo gem-300 install yus. Updating /service/yus/run to call sudo -u apache /usr/local/bin/ruby-300 /usr/local/ruby-3.0.0/bin/yusd Also updated /service/ch.oddb-meddata/run and /service/ch.oddb-refdata/run to use ruby-300 instead of ruby-272.

Discovered that running import_daily generates a OutOfMemory error. See https://github.com/zdavatz/oddb.org/issues/123.

Running sudo -u apache bundle-300 exec ruby-300 jobs/update_textinfo_swissmedicinfo pi to see, whether all patinfo can be parsed.

It looks like ODBA does not handle correctly empty arrays. Must update ydbi and odba for ruby 3.0, as there are various errors when running the tests.

Also ydocx tests have errors when running under Ruby 3.0. ydocs also should require 'rmagick' and not the out-dated 'RMagick', as gem name should always be lowercase.

It looks to me, that there are invalid fachinfos in our database, as on thinpower I get the following output

ch.oddb> fachinfos.find_all{ |x,y| y.iksnrs.size == 0 && y.localized_name == nil}.size
-> 239

Modifying in src/view/fachinfo.rb the method to_html to skip these invalid entries. Now update_textinfo_swissmedicinfo pi finishes, but memory useage is still very high (95% of my 16GB RAM, thinpower has 32GB). Adding some sleeps to allow odba cache cleanup to work. Also adding some debug info, to see how much memory is used to pinpoint the problem. Now the memory usage while updating the RSS stays at less than 3,5 GB, but prolongs the time needed to finish by around 400 seconds = 7 minutes.

Checking whether I can reduce this a little bit. Done. Running import_daily, then import_bsv_followers.

After many cleanups in htmlgrid, spec and unit tests the unit tests now pass all under Ruby 3.0. Under ruby head they fail, because there is not yet a nokogiri version. Under ruby 2.7 there are two obscure errors related to odba (I think).

There are stil some rspec failures. I do not want removed, because I am quite confident, that the expected behaviour is correct, but the test cases take at the moment too much time to fix them. I will document the failing tests here:

  1) ch.oddb.org desitin should find redirect an iphone to the mobile flavor
     Failure/Error: expect(iphone_browser.url).to match(/\/\/i\./) # this fails
  2) ch.oddb.org desitin should display the correct calculation for Bicalutamid Sandoz
     Failure/Error: tageskosten =  @browser.trs.find{|x| /^Tageskosten/.match(x.text)}.text
  3) ch.oddb.org desitin should have a working instant search for Inderal and going back
     Failure/Error: expect(inhalt).to match(/Preisvergleich f??r/i)
  4) ch.oddb.org desitin limiting the search to search_limitation_E using Holunder
     Failure/Error: expect(nr_unrestricted_first).to be > nr_restriced if must_be_greater
<..>
  1) ch.oddb.org should have a working RSS-feed /de/gcc/rss/channel/fachinfo
     Failure/Error: @browser.goto(link.href) if link
     Failure/Error: @browser.goto OddbUrl
<..>
  1) ch.oddb.org check pharmacy
     Failure/Error: @browser.link(visible_text: Fridolin).click
 1) ch.oddb.org should be possible to run grant_download oddb2.csv
<..>

Full Log is here: Attach:spec_2021_01_22_log.txt

But how should I fix the ATC-Importer. Last time I changed something substantial was were the following commits

view · edit · sidebar · attach · print · history
Page last modified on January 22, 2021, at 09:39 PM