view · edit · sidebar · attach · print · history

Index>

20160301-fix-import-daily

Summary

  • FI and changelog of newly imported FI has wrong language

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)

FI and changelog of newly imported FI has wrong language

I am a little bit confused. Last night the import daily completed successfully for FI and PI and completed in less than 30 minutes.

But running jobs/update_textinfo_swissmedicinfo --reparse --target=both 65856 65460 had an error when sorting the PI at

Error: ArgumentError
Message: comparison of Array with Array failed
Backtrace:
/var/www/oddb.org/src/util/language.rb:25:in `sort'
/var/www/oddb.org/src/util/language.rb:25:in `first'
/var/www/oddb.org/src/util/language.rb:32:in `description'
/var/www/oddb.org/src/util/language.rb:51:in `method_missing'
/var/www/oddb.org/src/util/persistence.rb:77:in `block in diff'
/var/www/oddb.org/src/util/persistence.rb:75:in `each'
/var/www/oddb.org/src/util/persistence.rb:75:in `diff'
/var/www/oddb.org/src/util/persistence.rb:276:in `issue_update'
/var/www/oddb.org/src/util/oddbapp.rb:138:in `block in update'
/var/www/oddb.org/src/util/failsafe.rb:10:in `call'
/var/www/oddb.org/src/util/failsafe.rb:10:in `failsafe'
/var/www/oddb.org/src/util/oddbapp.rb:136:in `update'
/var/www/oddb.org/src/util/oddbapp.rb:1645:in `update'
/var/www/oddb.org/src/plugin/text_info.rb:349:in `store_patinfo_for_all_packages'
/var/www/oddb.org/src/plugin/text_info.rb:390:in `block (2 levels) in update_patinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:389:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:389:in `block in update_patinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:380:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:380:in `update_patinfo_lang'
/var/www/oddb.org/src/plugin/text_info.rb:1270:in `parse_textinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1469:in `block in import_swissmedicinfo'

Checking the result of import_daily which reported eg. New/Updates PI from swissmedicinfo.ch Patinfo - DE - Aconitum-Homaccord, Tropfen["48561"]. Neither FI nor PI showed up on oddb-ci2.

For 63202 Ginkgo-Mepha the registration was created, but neither FI nor PI added correctly. Why?

Fixing the import. Rerunning import-daily and now I got a corectly looking http://oddb-ci2.dyndns.org/de/gcc/fachinfo/reg/63202. Changing the loop to always compare whether HTML-file is same or not (unless option reparse is given).

Got an error while updating the PI.

Error: NoMethodError
Message: undefined method `packages' for ["47783", "fi", "de"]:Array
Backtrace:
/var/www/oddb.org/src/model/registration.rb:241:in `block in packages'
/var/www/oddb.org/src/model/registration.rb:240:in `each'
/var/www/oddb.org/src/model/registration.rb:240:in `inject'
/var/www/oddb.org/src/model/registration.rb:240:in `packages'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/plugin/text_info.rb:1326:in `block (2 levels) in report_problematic_names'
/var/www/oddb.org/src/plugin/text_info.rb:1325:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1325:in `block in report_problematic_names'
/var/www/oddb.org/src/plugin/text_info.rb:1323:in `open'
/var/www/oddb.org/src/plugin/text_info.rb:1323:in `report_problematic_names'
/var/www/oddb.org/src/plugin/text_info.rb:1399:in `parse_aips_download'
/var/www/oddb.org/src/plugin/text_info.rb:1443:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:563:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:539:in `call'
/var/www/oddb.org/src/util/updater.rb:539:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:557:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:334:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:200:in `run'
jobs/import_daily:13:in `block in <module:Util>'

Looking for changelogs in bin/admin via

$f = registrations.values.find_all{|reg| reg.fachinfo && reg.fachinfo.de.change_log.size >0}
-> Array
ch.oddb> $f.size
-> 580
ch.oddb> $f.collect{|reg| reg.fachinfo.de.change_log.first.time}.uniq.sort.first
-> 2015-11-06
ch.oddb> $f.collect{|reg| reg.fachinfo.de.change_log.first.time}.uniq.sort.last
-> 2016-01-08

On thinpower this gives me the following result

ch.oddb> $f.size
-> 918
ch.oddb> $f.collect{|reg| reg.fachinfo.de.change_log.first.time}.uniq.sort.last
-> 2016-02-27
ch.oddb> $f.find{|reg| reg.fachinfo.de.change_log.first.time == Date.new(2016,02,27)}.iksnr
-> 08537
$f2 = $f.find_all{|reg| reg.fachinfo.de.change_log.first.time == Date.new(2016,02,27)}
ch.oddb> $f2.collect{|reg| reg.iksnr}
-> ["08537", "33670", "34383", "34385", "37551", "49526", "55263", "55649", "56285", "56552", "57244", "57555", "57673", "59455", "59876", "61337", "61344", "62181", "65314"]

On oddb-ci2 it did not update the FI 65314. Why? Using pry to debug the situation I found the missing reg.odba_store inside store_fachinfo.

Also fixed the error when patinfo got created.

Reloading database of January 11 and running import_daily Got no Changelogfor http://oddb-ci2.dyndns.org/de/gcc/fachinfo/reg/65314, but the changed line containing Aliskiren-haltigen Arzneimitteln as shown as changelog in http://ch.oddb.org/it/gcc/show/fachinfo/65314/diff/27.02.2016 is already present. Must reload the old database again, to check, whether this is really correct. The Change in ch.oddb.org seems to be quite old as it show at the end

-September 2013.
-Interne Versionsnummer: 1.4
+April 2015.
+Interne Versionsnummer: 2.2

With the dump of January 11 I did not have an FI for 65314.

Running import_daily got another weird error

Error: NotImplementedError
Message: method `method_missing' called on unexpected T_ZOMBIE object (0x0000009ef1ffb0 flags=0x103e klass=0x0)
Backtrace:
/var/www/oddb.org/src/plugin/text_info.rb:1460:in `block in import_swissmedicinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1453:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1453:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:560:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:536:in `call'
/var/www/oddb.org/src/util/updater.rb:536:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:554:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:333:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:200:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'
/var/www/oddb.org/log/job.pid is deleted
/var/www/oddb.org/src/plugin/text_info.rb:1460:in `block in import_swissmedicinfo': method `method_missing' called on unexpected T_ZOMBIE object (0x0000009ef1ffb0 flags=0x103e klass=0x0) (NotImplementedError)
        from /var/www/oddb.org/src/plugin/text_info.rb:1453:in `each'
        from /var/www/oddb.org/src/plugin/text_info.rb:1453:in `import_swissmedicinfo'
        from /var/www/oddb.org/src/util/updater.rb:560:in `block in update_notify_simple'
        from /var/www/oddb.org/src/util/updater.rb:536:in `call'
        from /var/www/oddb.org/src/util/updater.rb:536:in `wrap_update'
        from /var/www/oddb.org/src/util/updater.rb:554:in `update_notify_simple'
        from /var/www/oddb.org/src/util/updater.rb:333:in `update_textinfo_swissmedicinfo'
        from /var/www/oddb.org/src/util/updater.rb:200:in `run'
        from jobs/import_daily:13:in `block in <module:Util>'
        from /var/www/oddb.org/src/util/job.rb:40:in `call'
        from /var/www/oddb.org/src/util/job.rb:40:in `run'
        from jobs/import_daily:12:in `<module:Util>'
        from jobs/import_daily:11:in `<module:ODDB>'
        from jobs/import_daily:10:in `<main>'

I didn't now that we have zombies in oddb.org. I am getting another weird error. Found in dmesg an out of memory killer. May this had a destabilizing effect after trying to run the profiling last week.

Getting DB-Dump from yesterday. Then will reboot and reload the database. Okay. Looks better. I have now an FI for 65314. Running import_daily again.

Search for wrong changelogs in german/french of february 2016 via


$de = registrations.values.find_all{|reg| reg.fachinfo && reg.fachinfo.de.change_log.find{|ch| ch.time >= Date.new(2016,02,25)}}
$de.size
File.open('changelog.de.txt', 'w+') {|de| $de.each{|reg| de.puts "#{reg.iksnr}: #{reg.fachinfo.de.change_log.sort.last.diff.to_s.split("\n")[0..2]}" } }

$fr = registrations.values.find_all{|reg| reg.fachinfo && reg.fachinfo.fr.change_log.find{|ch| ch.time >= Date.new(2016,02,25)}}
$fr.size
File.open('changelog.fr.txt', 'w+') {|fr| $fr.each{|reg| fr.puts "#{reg.iksnr}: #{reg.fachinfo.fr.change_log.sort.last.diff.to_s.split("\n")[0..2]}" } }

See Attach:changelog.fr.txt and Attach:changelog.de.txt

Remarked,

  • 65479, 65460: FI french instead of german
  • 65383/65384 FI changed from german -> French ok

Reimporting 65479 and 65460 via jobs/update_textinfo_swissmedicinfo --reparse --target=both 65479 65460.

view · edit · sidebar · attach · print · history
Page last modified on March 01, 2016, at 03:58 PM