view · edit · sidebar · attach · print · history

< Niklaus.20220309-refdata-update | Index | Niklaus.20220121-ruby-31 >>

20220124-ruby-31-part-two

Summary

Commits

oddb.org:

sbsm:

Index:

Cleanup SBSM unit tests

Pushed a few commits to cleanup warnings, unit tests errors. Removed unused dependencies to watir and pry. Remarked that the ruby version 3.0 did not have a quote, which whould have meant it to equal to highest available 3.x version of ruby.

Lost an hour by checking that on thin-power and a personal debian machine, there are NO test errors when using ruby 3.0.3 and ruby 3.1.0 even when github/actions give failures. Do not know why this happens.

Port oddb.org to Ruby 3.1 part two

Getting a current dump from thinpower via scp ywesee@ywesee.com:/var/backup/thinpower/db/postgresql/January-2022/2022-01-22/22:00-postgresql_database-ch_oddb-backup.bz2 /var/backup/thinpower/db_dumps/pg-db-ch_oddb-backup.bz2 and /usr/local/bin/pg_create_db.sh after having stopped all services.

Using ruby-3.1 without the new

Restarting the services yus, migel and ch-oddb*. Patching src/plugin/text_info.rb to break on the error reported last friday. Restarting import_daily.

Fix Reparse Error

Parsing FI 36784 RHINOCAP® Kapseln: leads to errors. Looks like methods text and changelog throw errors, where they didn't before. But it is also a FI which has not been updated since 2007. Running sudo -u apache bundle-310 exec ruby-310 jobs/update_textinfo_swissmedicinfo --target=fi --reparse 36784 and placing a binding.irb to catch the error.

Used the new gem debug (replaces pry-byebug) to debug the problem. But for both FI and PI I got errors when calling odba_store. Tried various approaches without getting any success.

Storing the PI 67083 Vyndaqel failed. When I look at it on oddb-ci2 via the web I get an NoMethodError 'first' for PatinfoDocument

Deleting FI and PI of these two registrations to see, whether this are just some old hidden errors in the database. Calling bin/admin with

@h.oddb> reg = @system.registration('36784'); delete reg.fachinfo.pointer; update reg.pointer, :fachinfo => nil
-> #<ODDB::Registration:0x00007f2ecd26ea00>
ch.oddb> reg = @system.registration('67083'); delete reg.sequences.values.first.patinfo.pointer; update reg.sequences.values.first.patinfo = nil
-> wrong number of arguments (given 1, expected 2..3)
ch.oddb> reg = @system.registration('67083'); reg.sequences.values.first.patinfo
->
@

Restarting import sudo -u apache bundle-310 exec ruby-310 jobs/update_textinfo_swissmedicinfo --target=both --reparse 67083 36784

Now I am getting the following error ((recurses till the process gets killed)

@OdbaError
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:47:in `odba_receiver'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:27:in `is_a?'
/var/www/oddb.org/src/util/language.rb:31:in `description'
/var/www/oddb.org/src/util/language.rb:54:in `method_missing'
/var/www/oddb.org/src/util/language.rb:31:in `description'
/var/www/oddb.org/src/util/language.rb:54:in `method_missing'
/var/www/oddb.org/src/model/fachinfo.rb:79:in `localized_name'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:116:in `method_missing'
/var/www/oddb.org/src/view/rss/fachinfo.rb:85:in `block (2 levels) in to_html'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `each'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `block in to_html'
ODBA::Stub was unable to replace ODDB::SimpleLanguage::Descriptions#30593148 from ODDB::Fachinfo:#30593146. raise OdbaError
Descriptions.description error
/var/www/oddb.org/src/util/language.rb:29:in `description'
/var/www/oddb.org/src/util/language.rb:54:in `method_missing'
/var/www/oddb.org/src/util/language.rb:31:in `description'
/var/www/oddb.org/src/util/language.rb:54:in `method_missing'
/var/www/oddb.org/src/model/fachinfo.rb:79:in `localized_name'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:116:in `method_missing'
/var/www/oddb.org/src/view/rss/fachinfo.rb:85:in `block (2 levels) in to_html'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `each'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `block in to_html'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:439:in `make'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:403:in `make'
/usr/local/bin/ruby-310: line 3: 16019 Killed                  /usr/local/ruby-3.1.0/bin/ruby-310 $@

Looking at when the FI was parsed on thinpower I get

ywesee@thinpower /var/www/oddb.org $ find . -name "*36784*" | xargs ls -lrt
find: "./main": Keine Berechtigung
find: "./svc/oddb_google_crawler/log/supervise": Keine Berechtigung
-rw-r--r-- 1 apache users  75054 14. Dez 2011  ./doc/resources/patinfo/36784_01_1182520392.08234.pdf
-rw-r--r-- 1 apache users  67697 14. Dez 2011  ./doc/resources/patinfo/36784_01.pdf
-rw-r--r-- 1 apache users  75126 14. Dez 2011  ./doc/resources/patinfo/36784_01_1187354064.59001.pdf
-rw-r--r-- 1 apache apache 14487 25. Feb 2016  ./data/details/36784_fi_de.xml
-rw-r--r-- 1 apache apache 26280 25. Feb 2016  ./data/details/36784_fi_fr.xml
-rw-r--r-- 1 apache apache 11049 25. Feb 2016  ./data/details/36784_pi_de.xml
-rw-r--r-- 1 apache apache 11801 25. Feb 2016  ./data/details/36784_pi_fr.xml
-rw-r--r-- 1 apache apache 14276 25. Feb 2016  ./data/details/36784_pi_it.xml
-rw-r--r-- 1 apache apache 12894 20. Jan 07:11 ./data/details/36784_pi_de_Rhinocap___Kapseln_problem.xml
-rw-r--r-- 1 apache apache 13180 20. Jan 07:11 ./data/details/36784_pi_de_RHINOCAP__Kapseln_problem.xml
-rw-r--r-- 1 apache apache 12698 20. Jan 07:11 ./data/details/36784_pi_fr_RHINOCAP__Capsules_problem.xml
-rw-r--r-- 1 apache apache 14058 20. Jan 07:11 ./data/details/36784_pi_fr_Rhinocap__G_lules_problem.xml
-rw-r--r-- 1 apache apache 42423 20. Jan 07:11 ./data/details/36784_pi_it_RHINOCAP__capsule_problem.xml
-rw-r--r-- 1 apache apache 12574 20. Jan 07:11 ./data/details/36784_pi_it_Rhinocap___Capsule_problem.xml
-rw-r--r-- 1 apache apache 16576 25. Jan 07:09 ./data/details/36784_fi_de_RHINOCAP__Kapseln_problem.xml
-rw-r--r-- 1 apache apache 18601 25. Jan 07:09 ./data/details/36784_fi_de_Rhinocap___Kapseln_problem.xml
-rw-r--r-- 1 apache apache 20558 25. Jan 07:09 ./data/details/36784_fi_fr_Rhinocap___G_lules_problem.xml
-rw-r--r-- 1 apache apache 19219 25. Jan 07:09 ./data/details/36784_fi_fr_RHINOCAP__capsules_problem.xml
-rw-r--r-- 1 apache apache 18787 25. Jan 07:09 ./data/details/36784_fi_it.xml
-rw-r--r-- 1 apache apache 18787 25. Jan 07:36 ./data/details/36784_fi_it_Rhinocap___Capsule_problem.xml

And the PI for 67083

/var/www/oddb.org $ find . -name "*67083*" | xargs ls -lrt
find: "./main": Keine Berechtigung
find: "./svc/oddb_google_crawler/log/supervise": Keine Berechtigung
-rw-r--r-- 1 apache apache 184997  8. Apr 2020  ./data/details/67083_fi_de.xml
-rw-r--r-- 1 apache apache 187655  8. Apr 2020  ./data/details/67083_fi_fr.xml
-rw-r--r-- 1 apache apache 170143  8. Apr 2020  ./data/details/67083_fi_it.xml
-rw-r--r-- 1 apache apache  19261  8. Apr 2020  ./data/details/67083_pi_de.xml
-rw-r--r-- 1 apache apache  21107  8. Apr 2020  ./data/details/67083_pi_fr.xml
-rw-r--r-- 1 apache apache  18519  8. Apr 2020  ./data/details/67083_pi_it.xml
-rw-r--r-- 1 apache apache 184997  8. Apr 2020  ./data/details/67083_fi_de_Vyndaqel___Weichkapseln_problem.xml
-rw-r--r-- 1 apache apache 187655  8. Apr 2020  ./data/details/67083_fi_fr_Vyndaqel___Capsule_molle_problem.xml
-rw-r--r-- 1 apache apache 170143  8. Apr 2020  ./data/details/67083_fi_it_Vyndaqel___Capsule_molli_problem.xml
-rw-r--r-- 1 apache apache  19261 10. Apr 2020  ./data/details/67083_pi_de_Vyndaqel___Weichkapseln_problem.xml
-rw-r--r-- 1 apache apache  21107 10. Apr 2020  ./data/details/67083_pi_fr_Vyndaqel___Capsules_molles_problem.xml
-rw-r--r-- 1 apache apache  78448 10. Apr 2020  ./data/details/67083_pi_it_Vyndaqel___Capsule_molli_problem.xml
-rw-r--r-- 1 apache apache 185121 20. Dez 07:08 ./data/details/67083_fi_de_Vyndaqel_problem.xml
-rw-r--r-- 1 apache apache 187757 20. Dez 07:08 ./data/details/67083_fi_fr_Vyndaqel_problem.xml
-rw-r--r-- 1 apache apache 170288 20. Dez 07:08 ./data/details/67083_fi_it_Vyndaqel_problem.xml

After adding a few odba_store in @src/util/textinfo.rb@ with

diff --git a/src/plugin/text_info.rb b/src/plugin/text_info.rb
index cc4efa8f..9c3b6b33 100644
--- a/src/plugin/text_info.rb
+++ b/src/plugin/text_info.rb
@@ -320,18 +320,24 @@ module ODDB
msg += ' new patinfo'
else
package.patinfo = @app.create_patinfo
+        package.patinfo.odba_store
package.patinfo.descriptions[lang] = patinfo_lang
package.patinfo.odba_store
msg += ' created patinfo'
end
+      package.sequence.odba_store
package.odba_store
unless package.patinfo.descriptions.values.first.is_a?(ODDB::PatinfoDocument)
msg = "class #{package.patinfo.descriptions.values.first.class} is not a PatinfoDocument"
raise msg
end
# Update patinfo of sequence
+      package.patinfo.odba_store
+      package.sequence.odba_store
package.sequence.patinfo = package.patinfo  unless package.sequence.patinfo.object_id == package.patinfo.object_id
package.sequence.odba_store
+      package.patinfo.odba_store
+      package.odba_store
LogFile.debug "called odba_store #{msg}"
package.patinfo
end

Now the reparsing continues, but at the end I get the following recursion which results in a out of memory error

2022-02-22 10:52:13 +0100: /var/www/oddb.org/src/plugin/text_info.rb:341:in `store_package_patinfo': called odba_store 67518/01/001: fr Vyndaqel®, Capsules molles change_diff
2022-02-22 10:52:13 +0100: job is done. now postprocess works ...
OdbaError
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:47:in `odba_receiver'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:27:in `is_a?'
/var/www/oddb.org/src/util/language.rb:31:in `description'
/var/www/oddb.org/src/util/language.rb:54:in `method_missing'
/var/www/oddb.org/src/model/fachinfo.rb:79:in `localized_name'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:116:in `method_missing'
/var/www/oddb.org/src/view/rss/fachinfo.rb:85:in `block (2 levels) in to_html'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `each'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `block in to_html'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:439:in `make'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:403:in `make'
ODBA::Stub was unable to replace ODDB::SimpleLanguage::Descriptions#30593148 from ODDB::Fachinfo:#30593146. raise OdbaError
Descriptions.description error
/var/www/oddb.org/src/util/language.rb:29:in `description'
/var/www/oddb.org/src/util/language.rb:54:in `method_missing'
/var/www/oddb.org/src/model/fachinfo.rb:79:in `localized_name'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/odba-1.1.8/lib/odba/stub.rb:116:in `method_missing'
/var/www/oddb.org/src/view/rss/fachinfo.rb:85:in `block (2 levels) in to_html'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `each'
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `block in to_html'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:439:in `make'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:403:in `make'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker.rb:29:in `make'
/var/www/oddb.org/src/view/rss/fachinfo.rb:74:in `to_html'
OdbaError
/var/www/odd
<...>
/var/www/oddb.org/src/view/rss/fachinfo.rb:84:in `block in to_html'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:439:in `make'
/var/www/oddb.org/vendor/bundle/ruby/3.1.0/gems/rss-0.2.9/lib/rss/maker/base.rb:403:in `make'
[FATAL] failed to allocate memory

The patch (zdavatz#152) Fix reparsing textinfo for FI contains the following improvements:

  • src/plugin/text_info.rb contains (maybe more than necessary) calls to odba_store to avoid the reparse problem
  • Added a puts statement in jobs/update_textinfo_swissmedicinfo to clearly indicate, that everyhing went okay.
  • Removed a unneeded sleep in src/view/rss/fachinfo.rb and added some debug output to see how much memory was consumed$
  • src/plugin/plugin.rb contains a few log statement to see, when the RSS feeds are getting updated in each language
view · edit · sidebar · attach · print · history
Page last modified on February 24, 2022, at 10:07 AM