view · edit · sidebar · attach · print · history

Index>

20160323-textinfo-pi

Summary

  • Fix export therapeuticus
  • Fix update for patinfo
  • Import daily must show FI diffs

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)
  • BSV-Plugin does not send mail. Is it sending two mails at once?
  • remove obsolete pdf_patinfo in model, admin/views

Fix export therapeuticus

Must fix the following error seen today 

Error: ArgumentError
Message: comparison of NilClass with ODDB::Ean13 failed
Backtrace:
/var/www/oddb.org/src/plugin/csv_export.rb:161:in `sort_by'
/var/www/oddb.org/src/plugin/csv_export.rb:161:in `export_index_therapeuticus'
/var/www/oddb.org/src/util/exporter.rb:168:in `block in export_index_therapeuticus_csv'
/var/www/oddb.org/src/util/exporter.rb:343:in `call'
/var/www/oddb.org/src/util/exporter.rb:343:in `safe_export'
/var/www/oddb.org/src/util/exporter.rb:166:in `export_index_therapeuticus_csv'
/var/www/oddb.org/src/util/exporter.rb:73:in `run'
jobs/export_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/export_daily:12:in `<module:Util>'
jobs/export_daily:11:in `<module:ODDB>'
jobs/export_daily:10:in `<main>'

Problem probably caused because export daemon was not restarted. Restarting daemon and running jobs/mail_index_therapeuticus_csv. No. Problem still persists. Looks like yesterday I committed for this problem only the changed unit tests but not the real fix in src/plugin/csv_export.rb.

The export fails because we have 11 patinfo where the ikskey is nil 

ch.oddb> packages.find_all{|pac| pac.ikskey == nil}.each{|x| puts x.pointer}.size
-> 11
ch.oddb> packages.find_all{|pac| pac.ikskey == nil}.each{|x| puts x.pointer}
-> Array
# produces the following output
:!registration,49703!sequence,01!package,101.
:!registration,49703!sequence,01!package,102.
:!registration,49703!sequence,02!package,099.
:!registration,49703!sequence,02!package,100.
:!registration,49703!sequence,03!package,097.
:!registration,49703!sequence,03!package,098.
:!registration,50473!sequence,02!package,031.
:!registration,50473!sequence,02!package,058.
:!registration,55828!sequence,03!package,888.
:!registration,55829!sequence,01!package,010.
:!registration,55829!sequence,01!package,014.
# correctingg on oddb-ci2
ch.oddb> registration('55829').sequence('01').registration = registration('55829')
-> #<ODDB::Registration:0x007fdc188da328>
ch.oddb> registration('55829').odba_store
-> #<ODDB::Registration:0x007fdc188da328>
ch.oddb> registration('55829').sequence('01').package('010').iksnr
-> 55829
ch.oddb> registration('55828').sequence('03').package('888').sequence = registration('55828').sequence('03')
-> Ebixa 15 mg, Filmtabletten
ch.oddb> registration('55828').sequence('03').odba_store
-> Ebixa 15 mg, Filmtabletten
ch.oddb> packages.find_all{|pac| pac.iksnr == nil }.each{|x| puts x.pointer}
-> []
# 49703 has an array of registrations instead of a single element
ch.oddb> registration('49703').sequences.values.first.registration
-> [#<ODBA::Stub:70291238626720#34651062 @odba_class=Array @odba_container=70291238618860#34650836>, #<ODBA::Stub:70291238626060#34651063 @odba_class=Array @odba_container=70291238618860#34650836>]
ch.oddb> registration('49703').sequences.values.last.registration
-> [#<ODBA::Stub:70291238736440#34651066 @odba_class=Array @odba_container=70291238728200#34650838>, #<ODBA::Stub:70291238736040#34651067 @odba_class=Array @odba_container=70291238728200#34650838>]

We have bad elements in the database which must be corrected like this 

registration('55828').sequence('03').package('888').sequence = registration('55828').sequence('03')
registration('55828').sequence('03').odba_store
registration('55829').sequence('01').registration = registration('55829')
registration('55828').sequence('03').odba_store
registration('49703').sequences.values.each{|seq|  seq.registration = nil; seq.registration = registration('49703'); seq.odba_store }
registration('50473').sequences.values.each{|seq|  seq.registration = nil; seq.registration = registration('50473'); seq.odba_store }
# then everything is okay if the following statement returns an empty element
packages.find_all{|pac| pac.ikskey == nil }

Pushed commit Don't export nil packages. Pulled on thinpower and running jobs/mail_index_therapeuticus_csv.

Fix textinfo for patinfo

Banging my head why after reparsing 43788 not all PI are displayed correctly.

Verifying whether import is correct by issueing some bin/admin commands 

ch.oddb> registration('43788').active_packages.each{|x| puts "#{x.pointer}: fr #{x.patinfo.oid} #{x.patinfo.fr.name}"}
-> Array
ch.oddb> registration('43788').active_packages.each{|x| puts "#{x.pointer}: de #{x.patinfo.oid} #{x.patinfo.de.name}"}
-> Array
# generates out on oddbd
:!registration,43788!sequence,01!package,019.: fr 34665667 Tramal® Gouttes, solution à prendre
:!registration,43788!sequence,01!package,027.: fr 34665670 Tramal® Gouttes, solution à prendre
:!registration,43788!sequence,01!package,035.: fr 34665664 Tramal® Tropfen, Lösung zum Einnehmen
:!registration,43788!sequence,01!package,051.: fr 34665673 Tramal® Tropfen, Lösung zum Einnehmen mit Dosierpumpe
:!registration,43788!sequence,01!package,078.: fr 34665676 Tramal® Tropfen, Lösung zum Einnehmen mit Dosierpumpe
:!registration,43788!sequence,01!package,086.: fr 34665679 Tramal® Tropfen, Lösung zum Einnehmen mit Dosierpumpe
:!registration,43788!sequence,01!package,019.: de 34665667 Tramal® Gouttes, solution à prendre
:!registration,43788!sequence,01!package,027.: de 34665670 Tramal® Gouttes, solution à prendre
:!registration,43788!sequence,01!package,035.: de 34665664 Tramal® Tropfen, Lösung zum Einnehmen
:!registration,43788!sequence,01!package,051.: de 34665673 Tramal® Tropfen, Lösung zum Einnehmen mit Dosierpumpe
:!registration,43788!sequence,01!package,078.: de 34665676 Tramal® Tropfen, Lösung zum Einnehmen mit Dosierpumpe
:!registration,43788!sequence,01!package,086.: de 34665679 Tramal® Tropfen, Lösung zum Einnehmen mit Dosierpumpe

Some entries are not correct. Re-reading the code to see what could have caused this. Grepping for package 019 in my reparse log I find 

2016-03-22 23:21:26 +0100: /var/www/oddb.org/src/plugin/text_info.rb:265:in `store_patinfo_for_one_packages': 43788/01/unknown de :!registration,43788!sequence,01!package,019. 34665624 :!patinfo,34665624. ["Tramal® Tropfen, Lösung zum Einnehmen", "Was sind Tramal Tropfen Lösung zum Einnehmen und wann werden sie angewendet?"]
43788/01/unknown de :!registration,43788!sequence,01!package,019. 34665624 :!patinfo,34665624. ["Tramal® Tropfen, Lösung zum Einnehmen", "Was sind Tramal Tropfen Lösung zum Einnehmen und wann werden sie angewendet?"]
2016-03-22 23:21:27 +0100: /var/www/oddb.org/src/plugin/text_info.rb:265:in `store_patinfo_for_one_packages': 43788/01/unknown fr :!registration,43788!sequence,01!package,019. 34665633 :!patinfo,34665633. ["Tramal® Gouttes, solution à prendre", "Qu’est-ce que Tramal gouttes, solution à prendre et quand doit-il être utilisé?"]
43788/01/unknown fr :!registration,43788!sequence,01!package,019. 34665633 :!patinfo,34665633. ["Tramal® Gouttes, solution à prendre", "Qu’est-ce que Tramal gouttes, solution à prendre et quand doit-il être utilisé?"]

Looks that I did not conserve the correct german entry when I updated the the french. Probably it was empty.

Okay now the patinfos display nicely and correctly in german and french! Running import with less verbose debugging again before push commit.

Pushed commit Fix creating patinfo.pointer

Fixed missing french/english translations with commit Add french/english translation for complete consumer information

After running import_daily it looks as if the patinfo were recreated as it should be. Eg. Algifor does not appear in PI for Trulicity http://ch.oddb.org/de/gcc/patinfo/reg/65235/seq/01. To fix the problems with Tramal we need to run jobs/update_textinfo_swissmedicinfo --reparse --target=pi 43788 43787

Fix creating packages, parts and sequences in swissmedic

New packages, parts and sequences where created when running the swissmedic import without creating correctly their link to the outer. Fixing this with using the procedures from src/model like create_[sequence|package|part].

After resetting data/xls/Präparateliste-latest.xlsx, packungen.xls and XMLPublications (and reloading the database) to the state of february running jobs/import_swissmedic_only. Job is consuming about 6,8 GB of memory.

Seeing many lines like 2016-03-23 13:24:06 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb:990:in `update_package': create 00450/02/001 ptr :!create,:!registration,00450!sequence,02!package,001.. package #<ODDB::Package:0x000000a0db0>. Verifying whether this works fine via bin/admin 

ch.oddb> registration('00450').sequence('02').packages.keys
-> ["003", "004"]

Something went wrong! The new package is not stored correctly! Adding a pry statement and rerunning the import.

Import daily must show FI diffs

Why were the FI diffs not shown for Avamys? 

2016-03-23 07:21:04 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1155:in `extract_html': IKSNR 57968: File.size(/var/www/oddb.org/data/html/fi/de/Avamys_ Nasenspray.html) is 34747
2016-03-23 07:21:04 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1210:in `parse_textinfo': parse_textinfo 1210 /var/www/oddb.org/data/html/fi/de/Avamys_ Nasenspray.html does is not the same: ["57968"]
2016-03-23 07:21:04 +0100: /var/www/oddb.org/src/plugin/text_info.rb:232:in `update_fachinfo_lang': update_fachinfo_lang 57968 #<struct Struct::SwissmedicMetaInfo iksnr="57968", authNrs=["57968"], atcCode="R01AD12", title="Avamys\u00AE Nasenspray", authHolder="GlaxoSmithKline AG", substances="Fluticason furoat", type="fi", lang="de", informationUpdate="12.2015", refdata=nil, xml_file="/var/www/oddb.org/data/details/57968_fi_de_Avamys__Nasenspray_problem.xml", same_content_as_xml_file=false>  
2016-03-23 07:21:04 +0100: /var/www/oddb.org/src/plugin/text_info.rb:243:in `update_fachinfo_lang': update_fachinfo_lang Avamys® Nasenspray iksnr 57968 store_fachinfo {} ["de"] ATC R01AD12  
2016-03-23 07:21:04 +0100: /var/www/oddb.org/src/plugin/text_info.rb:148:in `store_fachinfo': store_fachinfo: 57968 ["de"]
2016-03-23 07:21:04 +0100: /var/www/oddb.org/src/plugin/text_info.rb:132:in `replace_textinfo': replace_textinfo updated :!registration,57968. :!registration,57968. type fachinfo :!fachinfo,29590624.
2016-03-23 07:21:04 +0100: /var/www/oddb.org/src/plugin/text_info.rb:184:in `ensure_correct_atc_code': ensure_correct_atc_code iksnr 57968 atcFromFI R01AD12 atcFromXml R01AD12 matched and found
2016-03-23 07:21:36 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1155:in `extract_html': IKSNR 57968: File.size(/var/www/oddb.org/data/html/fi/fr/Avamys_ Spray nasal.html) is 40687
2016-03-23 07:21:36 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1210:in `parse_textinfo': parse_textinfo 1210 /var/www/oddb.org/data/html/fi/fr/Avamys_ Spray nasal.html does is not the same: ["57968"]
2016-03-23 07:21:36 +0100: /var/www/oddb.org/src/plugin/text_info.rb:232:in `update_fachinfo_lang': update_fachinfo_lang 57968 #<struct Struct::SwissmedicMetaInfo iksnr="57968", authNrs=["57968"], atcCode="R01AD12", title="Avamys\u00AE Spray nasal", authHolder="GlaxoSmithKline AG", substances="Fluticasone furoate", type="fi", lang="fr", informationUpdate="12.2015", refdata=nil, xml_file="/var/www/oddb.org/data/details/57968_fi_fr_Avamys__Spray_nasal_problem.xml", same_content_as_xml_file=false>  
2016-03-23 07:21:36 +0100: /var/www/oddb.org/src/plugin/text_info.rb:243:in `update_fachinfo_lang': update_fachinfo_lang Avamys® Spray nasal iksnr 57968 store_fachinfo {} ["fr"] ATC R01AD12  
2016-03-23 07:21:36 +0100: /var/www/oddb.org/src/plugin/text_info.rb:148:in `store_fachinfo': store_fachinfo: 57968 ["fr"]
2016-03-23 07:21:38 +0100: /var/www/oddb.org/src/plugin/text_info.rb:132:in `replace_textinfo': replace_textinfo updated :!registration,57968. :!registration,57968. type fachinfo :!fachinfo,29590624.
2016-03-23 07:21:38 +0100: /var/www/oddb.org/src/plugin/text_info.rb:184:in `ensure_correct_atc_code': ensure_correct_atc_code iksnr 57968 atcFromFI R01AD12 atcFromXml R01AD12 matched and found
  57968 ["de"] Avamys
  57968 ["fr"] Avamys

Found the culprit. When reworking the loop to update the FI for all languages the call to add_change_log got forgotten.

Pushed commits

When running import_daily I got the following error, then an Out-Of-Memory which needed a reboot 

No need to add anything to /var/www/oddb.org/etc/barcode_to_text_info.yml
2016-03-23 22:01:19 CETlog notify Fach- und Patienteninfo Updates (swissmedicinfo.ch): sent mail
2016-03-23 22:01:19 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1418:in `import_swissmedicinfo': import_swissmedicinfo options  @options {:target=>:pi, :newest=>true}
2016-03-23 22:02:37 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1432:in `import_swissmedicinfo': Parsing @options[:newest] true true
2016-03-23 22:02:37 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1336:in `parse_aips_download': parse_aips_download with @options {:target=>:pi, :newest=>true}
2016-03-23 22:02:38 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1344:in `parse_aips_download': 2016-03-23 22:02:38 +0100: read 874194012 bytes
2016-03-23 22:03:25 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1312:in `report_problematic_names': 2016-03-23 22:03:25 +0100: Creating /var/www/oddb.org/log/problematic_fi_pi.lst
2016-03-23 22:03:35 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1332:in `report_problematic_names': created /var/www/oddb.org/log/problematic_fi_pi.lst
2016-03-23 22:03:35 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1435:in `import_swissmedicinfo': After parse_aips_download we have  13328 items to parse. Having 25160 meta items.
2016-03-23 22:07:00 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1407:in `get_swissmedicinfo_changed_items': get_swissmedicinfo_changed_items 0 @new_iksnrs
2016-03-23 22:07:00 CETlog notify Error: Fach- und Patienteninfo Updates (swissmedicinfo.ch): start outgoing process ["log"]. Must attach 0 files and 0 parts.
2016-03-23 22:07:00 CETUtil.send_mail list_and_recipients ["log"]
2016-03-23 22:07:00 CET Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: Fach- und Patienteninfo Updates (swissmedicinfo.ch) - 03/2016 size 1396 with 0 attachments. Plugi
Error: NoMethodError
Message: undefined method `values' for -9223372036854775808000000000:Bignum
Backtrace:
/var/www/oddb.org/src/model/registration.rb:241:in `block in packages'
/var/www/oddb.org/src/model/registration.rb:240:in `each'
/var/www/oddb.org/src/model/registration.rb:240:in `inject'
/var/www/oddb.org/src/model/registration.rb:240:in `packages'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/plugin/text_info.rb:1193:in `get_textinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1448:in `block in import_swissmedicinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1446:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1446:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:560:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:536:in `call'
/var/www/oddb.org/src/util/updater.rb:536:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:554:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:333:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:200:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

Looks like ruby got very confused.

The old text was calculated too late, therefore no changes were detected. Fixed with commit Fix FI compare as old_text was calculated too late

view · edit · sidebar · attach · print · history
Page last modified on March 24, 2016, at 09:39 AM