<< | Index | >>
---
I think I can reproduce this problem via bin/admin.
ch.oddb> indications.size
-> 3553
ch.oddb> res=[]; indications[0..-1].each{|x| res << x}; res.join(":").size
-> 3552
ch.oddb> indications.first.description
-> Zur Dämpfung des Hungergefühls
ch.oddb> indications.first.to_s
->
ch.oddb> indications.first.empty?
-> false
ch.oddb> indications.first.sequences
-> []
ch.oddb> indications.first.atc_classes.size
-> 1
Will recreate the index indication_index. Problem is that all indications return an empty string. Why?
Under ext/fiparse/indications.rb I find an implementation for indication(s) which depends on Rpdf2txt. Stopped this feature working when using textinfo_hpricot?
Found out, which function produce textinfo.
ch.oddb> registration('61565').fachinfo.interaction_text(:de)
-> Interaktionen Es wurden keine Wechselwirkungsstudien durchgefuehrt
ch.oddb> registration('61565').fachinfo.search_text(:de)
-> IndikationenAnwendungsmoeglichkeiten Triesence 40 mgml Injektionssuspension ist fuer die Visualisierung waehrend einer Vitrektomie indiziert
ch.oddb> registration('61565').fachinfo.unwanted_effect_text(:de).length
-> 2802
I think the following changes are needed:
bin/admin returns on my VM 537 for orphaned_fachinfos.size. Dito on thinpower. I did not find a way to find unused objects in the database.
Running jobs/rebuild_indices unwanted_effects_index_de (took about 13 minutes) and rerunning jobs/update_textinfo_swissmedicinfo --no-download --target=both --reparse 57435. This job started to use to much information, too. bin/oddbd console displayed warning ODBA::Stub was unable to replace ODDB::SimpleLanguage::Description when forcefully stopping the import.
Removing the unwanted_effects_index_de from etc/index_definitions.yaml and dropping the index via bin/admin ODBA.cache.drop_index unwanted_effects_index_de. Rerunning the import again. Now the memory begins to build up while updating unwanted_effects_index_fr.
Doing the same drop for unwanted_effects_index_fr. Then rerunning the import again. Now it completes and the fi shows correctly! Recreating the indices, git checkout etc/index_definitions.yaml. Then running jobs/rebuild_indices unwanted_effects_index_de. After about 23 minutes the rebuild job starts eating memory, too.
Want to look at the unwanted_text via bin/admin registration('57436').fachinfo.unwanted_effect_text(:de)
Added 57436 as a testcase to ext/fiparse/test/test_fachinfo_hpricot.rb. When trying to do add a debug info pp @fachinfo.unwanted_effects. I also consume too much memory and I get the following stack
^C/opt/src/oddb.org/src/model/text.rb:395:in `block in wrap': Interrupt
from /opt/src/oddb.org/src/model/text.rb:376:in `each'
from /opt/src/oddb.org/src/model/text.rb:376:in `wrap'
from /opt/src/oddb.org/src/model/text.rb:351:in `block (2 levels) in to_s'
from /opt/src/oddb.org/src/model/text.rb:350:in `collect'
from /opt/src/oddb.org/src/model/text.rb:350:in `block in to_s'
from /opt/src/oddb.org/src/model/text.rb:345:in `collect'
from /opt/src/oddb.org/src/model/text.rb:345:in `to_s'
from /opt/src/oddb.org/src/model/text.rb:430:in `block in to_s'
from /opt/src/oddb.org/src/model/text.rb:430:in `collect'
from /opt/src/oddb.org/src/model/text.rb:430:in `to_s'
from /opt/src/oddb.org/src/model/text.rb:489:in `block in to_s'
from /opt/src/oddb.org/src/model/text.rb:489:in `collect'
from /opt/src/oddb.org/src/model/text.rb:489:in `to_s'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:294:in `pretty_print'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:154:in `block in pp'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/prettyprint.rb:217:in `block (2 levels) in group'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/prettyprint.rb:243:in `nest'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/prettyprint.rb:216:in `block in group'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/prettyprint.rb:228:in `group_sub'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/prettyprint.rb:215:in `group'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:154:in `pp'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:77:in `block in pp'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:121:in `guard_inspect_key'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:77:in `pp'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:60:in `block in pp'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:59:in `each'
from /home/niklaus/.rvm/rubies/ruby-1.9.3-p448/lib/ruby/1.9.1/pp.rb:59:in `pp'
from ext/fiparse/test/test_fachinfo_hpricot.rb:911:in `test_unwanted_effects'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest/test.rb:110:in `block (3 levels) in run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest/test.rb:208:in `capture_exceptions'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest/test.rb:107:in `block (2 levels) in run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest/test.rb:260:in `time_it'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest/test.rb:106:in `block in run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:296:in `on_signal'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest/test.rb:280:in `with_info_handler'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest/test.rb:105:in `run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:268:in `block (2 levels) in run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:267:in `each'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:267:in `block in run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:296:in `on_signal'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:285:in `with_info_handler'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:266:in `run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:130:in `block in __run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:129:in `each'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:129:in `__run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:115:in `run'
from /home/niklaus/.rvm/gems/ruby-1.9.3-p448/gems/minitest-5.0.8/lib/minitest.rb:46:in `block in autorun'
Comparing this with stacktrace when running the import we see that no persistence is involved but that the method ODBA::Text::Chapter.to_s hangs. Now debugging the whole stuff with pry. The error occurs when trying to split the string '2%'.
The following diff fixes this problem
diff --git a/src/model/text.rb b/src/model/text.rb
index c2a0173..997e1fd 100644
--- a/src/model/text.rb
+++ b/src/model/text.rb
@@ -373,6 +373,7 @@ module ODDB
result = ''
left, right = nil
hyph = opts[:hyphenator]
+ return str if /\d+%$/.match(str)
str.split("\n").each do |line|
res = ''
words = line.split(' ')
Applying it to my VM. Recreating the indices and running the import.
http://oddb-ci.dyndns.org/de/gcc?search_query=Lebererkrankung&search_type=st_indication&flavor=gcc&language=de&event=search&state_id=58321500 display a lot of missing FI, e.g. 54410, 50732. This phenomena is no longer visible once the import completed.
The import finished fine. Pushed commit Fix problem unwanted_effects.to_s which consumed all memory
Dropping the database and restoring with the content of October first. Then running import_daily. Noted that while seening job is done. now postprocess works ... about 1,8 GB is consumed.
import_daily did not complete. Instead I got the following stack-trace by mail
Plugin: ODDB::SwissmedicPlugin Error: NoMethodError Message: undefined method `package' for "Tarka®":String Backtrace: /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/model/registration.rb:220:in `block in package' /var/www/oddb.org/src/model/registration.rb:219:in `each_value' /var/www/oddb.org/src/model/registration.rb:219:in `package' /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/plugin/swissmedic.rb:465:in `rows_diff' /var/www/.gem/ruby/1.9.1/bundler/gems/swissmedic-diff-0781cca7ac41/lib/swissmedic-diff.rb:133:in `block in diff' /var/www/.gem/ruby/1.9.1/bundler/gems/swissmedic-diff-0781cca7ac41/lib/swissmedic-diff.rb:303:in `block in each_valid_row' /usr/local/lib64/ruby/gems/1.9.1/gems/spreadsheet-0.9.0/lib/spreadsheet/worksheet.rb:144:in `block in each' /usr/local/lib64/ruby/gems/1.9.1/gems/spreadsheet-0.9.0/lib/spreadsheet/worksheet.rb:143:in `upto' /usr/local/lib64/ruby/gems/1.9.1/gems/spreadsheet-0.9.0/lib/spreadsheet/worksheet.rb:143:in `each' /usr/local/lib64/ruby/gems/1.9.1/gems/spreadsheet-0.9.0/lib/spreadsheet/excel/worksheet.rb:43:in `each' /var/www/.gem/ruby/1.9.1/bundler/gems/swissmedic-diff-0781cca7ac41/lib/swissmedic-diff.rb:294:in `each_valid_row' /var/www/.gem/ruby/1.9.1/bundler/gems/swissmedic-diff-0781cca7ac41/lib/swissmedic-diff.rb:107:in `diff' /var/www/oddb.org/src/plugin/swissmedic.rb:49:in `update' /var/www/oddb.org/src/util/updater.rb:387:in `block in update_swissmedic' /var/www/oddb.org/src/util/updater.rb:488:in `call' /var/www/oddb.org/src/util/updater.rb:488:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:385:in `update_swissmedic' /var/www/oddb.org/src/util/updater.rb:195:in `run' /var/www/oddb.org/jobs/import_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' /var/www/oddb.org/jobs/import_daily:12:in `<module:Util>' /var/www/oddb.org/jobs/import_daily:11:in `<module:ODDB>' /var/www/oddb.org/jobs/import_daily:10:in `<main>'
Remove all swissmedic-diff.rb, including old swissmedic_diff 0.1.3. Ran sudo gem install swissmedic-diff. Running jobs/import_swissmedic. At 13.30 the import is proceeding cleanly and already inserted all new registrations. Waiting for completion before committing and pushing the changes, as I had not yet received an email, which might cause an error, too.
After more than 10 hours the import finished. Had to fix an error in the report. Pushed commit Handle empty compositions
Reimported the database of October first. Rerun swissmedic. Completed successfully. See Attach:mail_swissmedic.txt