view · edit · sidebar · attach · print · history

Index>

20140701-Fix-invalid-byte-sequence-in-import-daily

Summary

  • Fix invalid byte sequence in US-ASCII when running import_daily

Commits

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • One unit-test for searchbar fails and might be a clue why searching does not work correctly.
  • Added two skip in test/test_plugin/rss.rb. Why does the mocking not work there anymore?
  • Make ext/swissindex/test/test_swissindex.rb and skipping tests in migel-gem pass.

---

Fix invalid byte sequence in US-ASCII when running import_daily

Got the following error mail

Plugin: ODDB::TextInfoPlugin
Error: ArgumentError
Message: invalid byte sequence in US-ASCII
Backtrace:
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:319:in `match'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:319:in `match'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:319:in `handle_text'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:214:in `handle_element'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:282:in `block in handle_all_children'
(druby://localhost:10002) /usr/local/lib/ruby/gems/1.9.1/gems/hpricot-0.8.6/lib/hpricot/traverse.rb:499:in `each'
(druby://localhost:10002) /usr/local/lib/ruby/gems/1.9.1/gems/hpricot-0.8.6/lib/hpricot/traverse.rb:499:in `each_child'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:281:in `handle_all_children'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:200:in `handle_element'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:43:in `chapter'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:89:in `block in extract'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:87:in `each'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/textinfo_hpricot.rb:87:in `extract'
(druby://localhost:10002) /var/www/oddb.org/ext/fiparse/src/fiparse.rb:201:in `parse_fachinfo_html'
(druby://localhost:10002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://localhost:10002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://localhost:10002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:10002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:10002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/text_info.rb:107:in `parse_fachinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1114:in `block in parse_and_update'
/var/www/oddb.org/src/plugin/text_info.rb:1078:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1078:in `parse_and_update'
/var/www/oddb.org/src/plugin/text_info.rb:1138:in `block (3 levels) in import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1137:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1137:in `block (2 levels) in import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1136:in `each_pair'
/var/www/oddb.org/src/plugin/text_info.rb:1136:in `block in import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1133:in `each_pair'
/var/www/oddb.org/src/plugin/text_info.rb:1133:in `import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1341:in `block in import_swissmedicinfo_by_index'
/var/www/oddb.org/src/plugin/text_info.rb:1340:in `each_pair'
/var/www/oddb.org/src/plugin/text_info.rb:1340:in `import_swissmedicinfo_by_index'
/var/www/oddb.org/src/plugin/text_info.rb:1395:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:525:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:501:in `call'
/var/www/oddb.org/src/util/updater.rb:501:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:519:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:308:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:193:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

Trying to get more info by adding some debug output and reproducing the problem on oddb-ci2 with the following steps

  • Getting yesterdays db_dump of thinpower
  • Re-Installing Ruby 1.9.3
  • Patch csv.rb
  • running import_daily

Got an error because flockhart plugin accessed a non existing table.aspx. Decided with Zeno to remove the flockhart plugin.

Pushed commits

Pulled them on thinpower.

Thinpower has another error

Plugin: ODDB::XlsExportPlugin
Error: NoMethodError
Message: undefined method `sequences' for "format_type":String
Backtrace:
(druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:119:in `comparables'
(druby://localhost:10005) /var/www/oddb.org/src/model/package.rb:214:in `comparables'
(druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/generics_xls.rb:96:in `block in export_generics'
(druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:175:in `each'
(druby://localhost:10005) /var/www/oddb.org/src/model/sequence.rb:175:in `each_package'
(druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
(druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:149:in `block in each_package'
(druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:148:in `each_value'
(druby://localhost:10005) /var/www/oddb.org/src/model/registration.rb:148:in `each_package'
(druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
(druby://localhost:10005) /var/www/oddb.org/src/util/oddbapp.rb:664:in `block in each_package'
(druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `each_value'
(druby://localhost:10005) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
(druby://localhost:10005) /var/www/oddb.org/src/util/oddbapp.rb:663:in `each_package'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/generics_xls.rb:94:in `export_generics'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:113:in `block in export_generics_xls'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `call'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `block in safe_export'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/tempfile.rb:316:in `open'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:285:in `safe_export'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:111:in `export_generics_xls'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/xls_export.rb:22:in `export_generics'
/var/www/oddb.org/src/util/updater.rb:119:in `block in export_generics_xls'
/var/www/oddb.org/src/util/updater.rb:501:in `call'
/var/www/oddb.org/src/util/updater.rb:501:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:117:in `export_generics_xls'
/var/www/oddb.org/src/util/updater.rb:271:in `update_bsv_followers'
/var/www/oddb.org/src/util/updater.rb:204:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

seen in log/oddb/debug/2014/07.log

2014-07-01 12:51:32 CEST Util.log_and_deliver_mail to=["Tim.Suter@just-medical.com", "matthijs.ouwerkerk@just-medical.com", "ngiger@ywesee.com", "ouwerkerk@bluewin.ch", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Med-Drugs - Swissmedic 05/2014 - SL 07/2014 size swissdrug update Swissmedic 05/2014 - SL 07/2014


Error registrations (Swissmedic Number):
17233
2014-07-01 12:51:36 CESTlog notify Med-Drugs: sent mail

I get the same error NoMethodError undefined method `description' for "format_type":String when trying to display the drug with iksnr 17233 via the web interface. bin/oddbd displays even more info

NoMethodError
undefined method `description' for "format_type":String
/var/www/oddb.org/src/view/additional_information.rb:81:in `atc_description'
/var/www/oddb.org/src/view/drugs/package.rb:175:in `atc_class'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:68:in `create'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:286:in `compose_component'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:211:in `block in compose'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `compose'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init'
/var/www/oddb.org/src/view/drugs/package.rb:170:in `init'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:61:in `new'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:61:in `create'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:286:in `compose_component'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:211:in `block in compose'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `compose'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init'
/var/www/oddb.org/src/view/drugs/package.rb:301:in `init'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize'
/var/www/oddb.org/src/view/publictemplate.rb:63:in `new'
/var/www/oddb.org/src/view/publictemplate.rb:63:in `content'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:68:in `create'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:286:in `compose_component'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:211:in `block in compose'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `compose'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init'
/var/www/oddb.org/src/view/publictemplate.rb:60:in `init'
/var/www/oddb.org/src/view/privatetemplate.rb:17:in `init'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/state.rb:245:in `new'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/state.rb:245:in `view'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/state.rb:181:in `to_html'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/session.rb:447:in `to_html'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/session.rb:173:in `block in drb_process'
<internal:prelude>:10:in `synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/session.rb:171:in `drb_process'
/usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
/usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
/usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
/usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
/usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
ODDB::View::Drugs::PackageInnerComposite::COMPONENTS[[0, 3, 0]] in create(atc_class)
ODDB::View::Drugs::PackageComposite::COMPONENTS[[0, 1]] in create(ODDB::View::Drugs::PackageInnerComposite)
ODDB::View::Drugs::Package::COMPONENTS[[0, 3]] in create(content)
error in SBSM::Session#http_headers: /de/gcc
NoMethodError
undefined method `description' for "format_type":String
/var/www/oddb.org/src/view/additional_information.rb:81:in `atc_description'
/var/www/oddb.org/src/view/drugs/package.rb:175:in `atc_class'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:68:in `create'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:286:in `compose_component'
/usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:211:in `block in compose'

Using bin/admin shows a very curious result

ch.oddb> registration('17233').sequence('02').package('001').atc_class
-> format_type

Found it as a change /var/www/oddb.org/data/xls/Präparateliste-latest.xlsx with the following details "17233"=>[:name_base, :index_therapeuticus, :production_science, :registration_date, :expiry_date, :ikscd, :size, :unit, :ikscat, :substances, :composition, :indication_registration]. It looks that our handling for this kind of change is wrong! Reparsing FI/PI for 17233. Did not help.

Using bin/admin found that we have exactly two packages (both belonging to IKSNR 17233) where the atc_class is a String.

$all = []; packages.collect {|x| $all << x if x.atc_class.is_a?(String)}; $all.size
-> 2
ch.oddb> $all.first
-> #<ODDB::Package:0x0000000a3b1140>
ch.oddb> $all.first.iksnr
-> 17233
ch.oddb> $all.first.ikscd
-> 068
ch.oddb> $all.last.iksnr
-> 17233
ch.oddb> $all.last.ikscd
-> 001
ch.oddb> $all = []; packages.collect {|x| $all << x unless x.atc_class.is_a?(ODDB::AtcClass)}; $all.size
-> 43
ch.oddb> $classes = $all.collect{ |x| x.atc_class.class.to_s }.sort.uniq
-> ["NilClass", "String"]

And we have 41 where the atc_class is nil.

Discovered that when calling jobs/update_textinfo_swissmedicinfo with an IKSNR which does not yet have a registration, that it will fail, because of a missing app.registrations.odba_store. But somehow, also the package name does not get updated correctly, probably because name_base is nil. Why?

The patinfo works and is shown under http://oddb-ci2.dyndns.org/de/gcc/patinfo/reg/17233/seq/00. Importing the fachinfo still fails.

view · edit · sidebar · attach · print · history
Page last modified on July 01, 2014, at 05:33 PM