oddb.org branch ruby_21x
odba
---
As running jobs/update_textinfo_swissmedicinfo --no-download --target=both 54842 --reparse fails under Ruby 1.9.3 on thinpower I must fix this problem first. Therefore switching back to branch master and reinstall ruby 1.9.3 on oddb-ci2. Running import for fi and pi and both, to see whether which combinations fail. Failing are :both and :fi whereas parsing pi only passes.
But running this particulare unit-test does not work
bundle exec ruby ext/fiparse/test/test_fachinfo_hpricot.rb
/home/niklaus/.rvm/gems/ruby-1.9.3-p484/bundler/gems/ruby-dbi-3b9421fb94f4/lib/dbi/types.rb:2:in `require': no implicit conversion from nil to integer (TypeError)
from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/bundler/gems/ruby-dbi-3b9421fb94f4/lib/dbi/types.rb:2:in `<top (required)>'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/bundler/gems/ruby-dbi-3b9421fb94f4/lib/dbi.rb:53:in `require'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/bundler/gems/ruby-dbi-3b9421fb94f4/lib/dbi.rb:53:in `<top (required)>'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/odba-1.1.0/lib/odba/storage.rb:8:in `require'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/odba-1.1.0/lib/odba/storage.rb:8:in `<top (required)>'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/odba-1.1.0/lib/odba.rb:4:in `require'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p484/gems/odba-1.1.0/lib/odba.rb:4:in `<top (required)>'
from /opt/src/oddb.org/src/util/persistence.rb:6:in `require'
from /opt/src/oddb.org/src/util/persistence.rb:6:in `<top (required)>'
from /opt/src/oddb.org/src/model/fachinfo.rb:9:in `require'
from /opt/src/oddb.org/src/model/fachinfo.rb:9:in `<top (required)>'
from /opt/src/oddb.org/ext/fiparse/src/fachinfo_hpricot.rb:7:in `require'
from /opt/src/oddb.org/ext/fiparse/src/fachinfo_hpricot.rb:7:in `<top (required)>'
from ext/fiparse/test/test_fachinfo_hpricot.rb:14:in `require'
from ext/fiparse/test/test_fachinfo_hpricot.rb:14:in `<main>'
Why? Will continue next monday.
As seen in yesterday's blog the import-daily had memory problems under Ruby 2.1.2, too.
Therefore I will use the lazy sax-parser too and try to minimize it's impact, but first I pull in yesterday dump of the database and restarted import_daily at 09.16.
Found a cleaner way to avoid an error in odba_add_reference(object) by adding to /usr/local/lib64/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache_entry.rb
def odba_add_reference(object)
return unless object.respond_to?(:odba_id)
This is at least something I could justify a patch, see commit Added fix for cache_entry.rb (no respond_to? odba_id) for Ruby 2.1.x
Made tests pass on travis with commits
When runn import_daily I remarked several (7) errors of the kind
NoMethodError: undefined method `fachinfo' for nil:NilClass when updating index 'unwanted_effects_index_fr' with a ODDB::Sequence ["(eval):2:in `block in proc_instance_origin'", "/usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:202:in `call'", "/usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:202:in `update_target'", "/usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:161:in `update'"] [...]
import_daily crashed with a ruby exception. See Attach:import_daily_2014_06_24_1.txt.
Therefore reverted sax-parser with commit Reverted sax-parser and ported another small fix from the master branch https://github.com/ngiger/oddb.org/commit/540f5a11de6e65be6bbebafbb028e6450619477c Avoid nil ptr.target also using old patch for cache_entry.rb. Reloaded db-dump from thinpower of yesterday, coied old *latest* files from thipower and restarted import_daily.
Import-daily hang. Sending a kill command generated a mail with the following backtrace
Plugin: ODDB::TextInfoPlugin Error: SignalException Message: SIGTERM Backtrace: /var/www/oddb.org/src/model/text.rb:382:in `block in wrap' /var/www/oddb.org/src/model/text.rb:377:in `each' /var/www/oddb.org/src/model/text.rb:377:in `wrap' /var/www/oddb.org/src/model/text.rb:351:in `block (2 levels) in to_s' /var/www/oddb.org/src/model/text.rb:350:in `collect' /var/www/oddb.org/src/model/text.rb:350:in `block in to_s' /var/www/oddb.org/src/model/text.rb:345:in `collect' /var/www/oddb.org/src/model/text.rb:345:in `to_s' /var/www/oddb.org/src/model/text.rb:431:in `block in to_s' /var/www/oddb.org/src/model/text.rb:431:in `collect' /var/www/oddb.org/src/model/text.rb:431:in `to_s' /var/www/oddb.org/src/model/text.rb:490:in `block in to_s' /var/www/oddb.org/src/model/text.rb:490:in `collect' /var/www/oddb.org/src/model/text.rb:490:in `to_s' /var/www/oddb.org/src/model/fachinfo.rb:116:in `unwanted_effect_text' (eval):3:in `block in proc_resolve_search_term' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:146:in `call' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:146:in `search_term' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:149:in `search_terms' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:174:in `update_origin' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/index.rb:163:in `update' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:590:in `block in update_indices' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:589:in `each' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:589:in `update_indices' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/cache.rb:515:in `store' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:274:in `odba_isolated_store' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:395:in `block in odba_store_unsaved' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:392:in `each' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:392:in `odba_store_unsaved' /usr/local/lib/ruby/gems/2.1.0/gems/odba-1.1.0/lib/odba/persistable.rb:380:in `odba_store' /var/www/oddb.org/src/util/persistence.rb:279:in `issue_update' /var/www/oddb.org/src/util/oddbapp.rb:133:in `block in update' /var/www/oddb.org/src/util/failsafe.rb:10:in `call' /var/www/oddb.org/src/util/failsafe.rb:10:in `failsafe' /var/www/oddb.org/src/util/oddbapp.rb:131:in `update' /var/www/oddb.org/src/util/oddbapp.rb:1575:in `update' /var/www/oddb.org/src/plugin/text_info.rb:139:in `store_fachinfo' /var/www/oddb.org/src/plugin/text_info.rb:182:in `block in update_fachinfo' /var/www/oddb.org/src/plugin/text_info.rb:175:in `each' /var/www/oddb.org/src/plugin/text_info.rb:175:in `update_fachinfo' /var/www/oddb.org/src/plugin/text_info.rb:1130:in `parse_and_update' /var/www/oddb.org/src/plugin/text_info.rb:1140:in `block (3 levels) in import_info' /var/www/oddb.org/src/plugin/text_info.rb:1139:in `each' /var/www/oddb.org/src/plugin/text_info.rb:1139:in `block (2 levels) in import_info' /var/www/oddb.org/src/plugin/text_info.rb:1138:in `each_pair' /var/www/oddb.org/src/plugin/text_info.rb:1138:in `block in import_info' /var/www/oddb.org/src/plugin/text_info.rb:1135:in `each_pair' /var/www/oddb.org/src/plugin/text_info.rb:1135:in `import_info' /var/www/oddb.org/src/plugin/text_info.rb:1343:in `block in import_swissmedicinfo_by_index' /var/www/oddb.org/src/plugin/text_info.rb:1342:in `each_pair' /var/www/oddb.org/src/plugin/text_info.rb:1342:in `import_swissmedicinfo_by_index' /var/www/oddb.org/src/plugin/text_info.rb:1397:in `import_swissmedicinfo' /var/www/oddb.org/src/util/updater.rb:525:in `block in update_notify_simple' /var/www/oddb.org/src/util/updater.rb:501:in `call' /var/www/oddb.org/src/util/updater.rb:501:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:519:in `update_notify_simple' /var/www/oddb.org/src/util/updater.rb:308:in `update_textinfo_swissmedicinfo' /var/www/oddb.org/src/util/updater.rb:193:in `run' jobs/import_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:42:in `call' /var/www/oddb.org/src/util/job.rb:42:in `run' jobs/import_daily:12:in `<module:Util>' jobs/import_daily:11:in `<module:ODDB>' jobs/import_daily:10:in `<main>'
I think we might really have a problem while trying to persist our changes to the database. Possible causes might be:
It think I will first try add to src/util/job.rb, drop the database again and restart the import_daily. Now the import_daily finishes without problem and a lot faster. But I think the problem is that we should force the YAML::ENGINE.yamler to "syck" at the very beginning of each file to run in bin and jobs. Okay, taking a close look at my changes I see that calling YAML::ENGINE.yamler = "syck" fails withand (overlooked by me) warnint syck has been removed, psych is used instead if there was no preceding require 'syck'. See this code
2.1.2 :001 > require 'yaml' => true 2.1.2 :002 > YAML::ENGINE.yamler => "psych" 2.1.2 :003 > YAML::ENGINE.yamler = "syck" syck has been removed, psych is used instead => "syck" 2.1.2 :004 > YAML::ENGINE.yamler => "psych" 2.1.2 :005 > require 'syck' => true 2.1.2 :006 > YAML::ENGINE.yamler = "syck" => "syck" 2.1.2 :007 > YAML::ENGINE.yamler => "syck" 2.1.2 :008 >
Pushed the following commits:
Dropping and reimporting the database again. Then restarting the daily_import. And this time it hangs again for after the line 2014-06-24 15:37:12 +0200: update_fachinfo CoAprovelŪ 150/12,5; 300/12,5; 300/25 iksnr 54842 store_fachinfo {} CoAprovel is found via import_swissmedicinfo_by_index. And again we get the same stack-trace as above when storing unwanted_effect_text. Restarting import_daily another time (without restart of oddbd). This time we have no problem. Will try import only fi and pi for this IKSNR (jobs/update_textinfo_swissmedicinfo --no-download --target=both 54842 --reparse). Fails with the same stack-trace. Now doing the same import on thinpower. Deleting fachinfo via bin/admin and restarting textinfo_reparse. Hangs
Must also have look at why at travis the rake quanty fails for travis. Trying it with using bundle exec rake quanty, see commit Always use bundle exec for travis cmds