I still have problems, for about 104 wirstofffe which show up on each import. They are created, but not correctly found. Using pry to debug this situation when running jobs/import_swissmedic_only update_compositions 65201. Found out that the dose for chemicals is not checked correctly.
[23] pry(#<ODDB::SwissmedicPlugin>)> agent.chemical_dose.to_s => "96 " [24] pry(#<ODDB::SwissmedicPlugin>)> substance.chemical_dose => "96 %"
It looks as I did not find (or store) the correct chemical_dose. Looks like it is a difficult to parse composition proteinorum plasmatis humani solutio 50 mg corresp. immunoglobulinum humanum normale min. 96 % et hepatitidis B virus antigenum min. 50 U.I., glycinum, residui: immunoglobulinum A max. 2 mg, aqua q.s. ad solutionem pro 1 ml.
This seems to work. But now I have probelm with IKSNR 00613. Reloading the database and running jobs/import_swissmedic_only update_compositions 00613
The first import says
Total time to update: 1.00 [m] Checked compositions: 7 Deleted compositions: 2 Updated agents: 0 New agents: 6
The second import says
Report 30.03.2015 Total time to update: 1.00 [m] Checked compositions: 6 Deleted compositions: 1 Updated agents: 0 New agents: 6
The Questions is why I delete one composition and add 6 new agents when running the second time.
The full log of the second run is
opts == {:fix_galenic_form=>false, :iksnrs=>[613], :update_compositions=>true}
2015-03-30 14:11:21 +0200: SwissmedicPlugin @index_url https://www.swissmedic.ch/arzneimittel/00156/00221/00222/00230/index.html?lang=de
update_swissmedic args [{:fix_galenic_form=>false, :iksnrs=>[613], :update_compositions=>true}]
2015-03-30 14:11:23 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:360 skip writing /var/www/oddb.org/data/xls/Packungen-2015.03.30.xlsx as /var/www/oddb.org/data/xls/Packungen-latest.xlsx is 2562552 bytes. Returning latest
2015-03-30 14:11:23 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:89 opts {:fix_galenic_form=>false, :iksnrs=>[613], :update_compositions=>true} @update_comps true update target nilLatest /var/www/oddb.org/data/xls/Packungen-latest.xlsx 2562552 bytes
2015-03-30 14:11:23 +0200: file2open /var/www/oddb.org/data/xls/Packungen-latest.xlsx 2562552 bytes
2015-03-30 14:12:20 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:590 update_active_agent update 00613/01 from creator 2 substances '' ptr :!create,:!registration,00613!sequence,01!composition,33020118!active_agent,Toxoidum Diphtheriae.. label nil 2 args {:dose=>Quanty(30,'UI/ml'), :substance=>24362395} chemical_dose
2015-03-30 14:12:28 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:590 update_active_agent update 00613/01 from creator 4 substances '' ptr :!create,:!registration,00613!sequence,01!composition,33020118!active_agent,Toxoidum Tetani 40 U%.i%.%, Toxoidum Pertussis 25 µg Et Haemagglutininum Filamentosum 25 µg%, Virus Poliomyelitis Typus 1 Inactivatum (d-antigen).. label nil 2 args {:dose=>Quanty(40,'U/ml'), :substance=>33020104} chemical_dose
2015-03-30 14:12:29 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:590 update_active_agent update 00613/01 from creator 6 substances '' ptr :!create,:!registration,00613!sequence,01!composition,33020118!active_agent,Virus Poliomyelitis Typus 2 Inactivatum (d-antigen).. label nil 2 args {:dose=>Quanty(8,'U/ml'), :substance=>24362637} chemical_dose
2015-03-30 14:12:29 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:590 update_active_agent update 00613/01 from creator 8 substances '' ptr :!create,:!registration,00613!sequence,01!composition,33020118!active_agent,Virus Poliomyelitis Typus 3 Inactivatum (d-antigen).. label nil 2 args {:dose=>Quanty(32,'U/ml'), :substance=>24362641} chemical_dose
2015-03-30 14:12:30 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:590 update_active_agent update 00613/01 from creator 10 substances '' ptr :!create,:!registration,00613!sequence,01!composition,33020118!active_agent,Formaldehydum.. label nil 2 args {:dose=>Quanty(10,'µg/ml'), :substance=>3029} chemical_dose
2015-03-30 14:12:31 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:590 update_active_agent update 00613/01 from creator 12 substances '' ptr :!create,:!registration,00613!sequence,01!composition,33020118!active_agent,Phenoxyethanolum.. label nil 2 args {:dose=>Quanty(2.5,'µl/ml'), :substance=>5485} chemical_dose
2015-03-30 14:12:31 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:657 iksnr 00613 seqnr 01 comp_idx 1 delete_composition :!registration,00613!sequence,01!composition,33020118. comp.oid 33020118 Toxoidum Diphtheriae 30 UI/ml, Toxoidum Diphtheriae 30 UI/ml, Toxoidum Tetani 40 u.I., Toxoidum Pertussis 25 µg et Haemagglutininum Filamentosum 25 µg, Virus Poliomyelitis Typus 1 Inactivatum (D-Antigen) 40 U/ml, Toxoidum Tetani 40 u.I., Toxoidum Pertussis 25 µg et Haemagglutininum Filamentosum 25 µg, Virus Poliomyelitis Typus 1 Inactivatum (D-Antigen) 40 U/ml, Virus Poliomyelitis Typus 2 Inactivatum (D-Antigen) 8 U/ml, Virus Poliomyelitis Typus 2 Inactivatum (D-Antigen) 8 U/ml, Virus Poliomyelitis Typus 3 Inactivatum (D-Antigen) 32 U/ml, Virus Poliomyelitis Typus 3 Inactivatum (D-Antigen) 32 U/ml, Formaldehydum 10 µg/ml, Formaldehydum 10 µg/ml, Phenoxyethanolum 2.5 µl/ml, Phenoxyethanolum 2.5 µl/ml
/var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Swissmedic XLS - 02/2015 size 153 with 0 attachments. ODDB::SwissmedicPlugin - Report 30.03.2015
Total time to update: 1.00 [m]
Checked compositions: 6
Deleted compositions: 1
Updated agents: 0
New agents: 6
After running the import I get the following error when visiting http://oddb-ci2.dyndns.org/de/gcc/drug/reg/00613/seq/01
error in SBSM::Session#to_html: /de/gcc/drug/reg/00613/seq/01 NoMethodError undefined method `resolve' for nil:NilClass /var/www/oddb.org/src/util/persistence.rb:92:in `parent' /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/view/admin/sequence.rb:55:in `_compose_footer' /var/www/oddb.org/src/view/admin/sequence.rb:114:in `compose_footer' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/list.rb:56:in `compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/list.rb:129:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize' /var/www/oddb.org/src/view/admin/sequence.rb:209:in `new' /var/www/oddb.org/src/view/admin/sequence.rb:209:in `composition' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:143:in `create' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divcomposite.rb:33:in `block in compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divcomposite.rb:13:in `each' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divcomposite.rb:13:in `compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divlist.rb:12:in `block in compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divlist.rb:10:in `each' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divlist.rb:10:in `each_with_index' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divlist.rb:10:in `compose' /var/www/oddb.org/src/view/admin/sequence.rb:204:in `compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:140:in `new' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:140:in `create' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divcomposite.rb:33:in `block in compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divcomposite.rb:13:in `each' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/divcomposite.rb:13:in `compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize' /var/www/oddb.org/src/view/admin/sequence.rb:672:in `new' /var/www/oddb.org/src/view/admin/sequence.rb:672:in `compositions' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:68:in `create' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:286:in `compose_component' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:211:in `block in compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `each' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/form.rb:82:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:61:in `new' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:61:in `create' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:286:in `compose_component' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:211:in `block in compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `each' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init' /var/www/oddb.org/src/view/admin/sequence.rb:621:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize' /var/www/oddb.org/src/view/publictemplate.rb:62:in `new' /var/www/oddb.org/src/view/publictemplate.rb:62:in `content' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:68:in `create' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:286:in `compose_component' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:211:in `block in compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `each' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:205:in `compose' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/composite.rb:57:in `init' /var/www/oddb.org/src/view/publictemplate.rb:59:in `init' /var/www/oddb.org/src/view/privatetemplate.rb:17:in `init' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/component.rb:139:in `initialize' /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.5/lib/sbsm/state.rb:245:in `new' /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.5/lib/sbsm/state.rb:245:in `view' /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.5/lib/sbsm/state.rb:181:in `to_html' /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.5/lib/sbsm/session.rb:448:in `to_html' /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.5/lib/sbsm/session.rb:173:in `block in drb_process' <internal:prelude>:10:in `synchronize' /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.5/lib/sbsm/session.rb:171:in `drb_process' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop' ODDB::View::Admin::RootCompositionList::COMPONENTS[[0, 0]] in create(composition) ODDB::View::Admin::RootCompositions::COMPONENTS[[0, 0]] in create(ODDB::View::Admin::RootCompositionList) ODDB::View::Admin::RootSequenceForm::COMPONENTS[[0, 4]] in create(compositions) ODDB::View::Admin::RootSequenceComposite::COMPONENTS[[0, 1]] in create(ODDB::View::Admin::RootSequenceForm) ODDB::View::Admin::RootSequence::COMPONENTS[[0, 3]] in create(content) error in SBSM::Session#http_headers: /de/gcc/drug/reg/00613/seq/01 NoMethodError undefined method `resolve' for nil:NilClass /var/www/oddb.org/src/util/persistence.rb:92:in `parent' /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/view/admin/sequence.rb:55:in `_compose_footer' /var/www/oddb.org/src/view/admin/sequence.rb:114:in `compose_footer' /usr/local/lib/ruby/gems/1.9.1/gems/htmlgrid-1.0.6/lib/htmlgrid/list.rb:56:in `compose'
Found the subtle differences when between active_agents and substances for 00631/01 on thinpower via bin/admin
ch.oddb> registration('00613').sequence('01').substances.first.to_s
-> Toxoidum Diphtheriae
ch.oddb> registration('00613').sequence('01').active_agents.first.to_s
-> Toxoidum Diphtheriae 30 UI/ml
ch.oddb> registration('00613').sequence('01').active_agents.first.oid
-> 24362649
ch.oddb> registration('00613').sequence('01').substances.first.oid
-> 24362395
Reloading database and breaking when adding the first composition. Using pry I find that the new compostion has as label nil instead of 'I'.
26] pry(#<ODDB::SwissmedicPlugin>)> seq.compositions[0].label => "I" [27] pry(#<ODDB::SwissmedicPlugin>)> comp.label => nil [28] pry(#<ODDB::SwissmedicPlugin>)> comp.source => "I) DTPa-IPV-Komponente (Suspension): toxoidum diphtheriae 30 U.I., toxoidum tetani 40 U.I., toxoidum pertussis 25 µg et haemagglutininum filamentosum 25 µg, virus poliomyelitis typus 1 inactivatum (D-Antigen) 40 U., virus poliomyelitis typus 2 inactivatum (D-Antigen) 8 U., virus poliomyelitis typus 3 inactivatum (D-Antigen) 32 U., aluminium ut aluminii hydroxidum hydricum ad adsorptionem, formaldehydum 10 µg, conserv.: phenoxyethanolum 2.5 µl, residui: neomycinum, streptomycinum, polymyxini B sulfas, medium199, aqua q.s. ad suspensionem pro 0.5 ml." [29] pry(#<ODDB::SwissmedicPlugin>)> seq.compositions[0].source => "I) DTPa-IPV-Komponente (Suspension): toxoidum diphtheriae 30 U.I., toxoidum tetani 40 U.I., toxoidum pertussis 25 µg et haemagglutininum filamentosum 25 µg, virus poliomyelitis typus 1 inactivatum (D-Antigen) 40 U., virus poliomyelitis typus 2 inactivatum (D-Antigen) 8 U., virus poliomyelitis typus 3 inactivatum (D-Antigen) 32 U., aluminium ut aluminii hydroxidum hydricum ad adsorptionem, formaldehydum 10 µg, conserv.: phenoxyethanolum 2.5 µl, residui: neomycinum, streptomycinum, polymyxini B sulfas, medium199, aqua q.s. ad suspensionem pro 0.5 ml."
Working on fixes for handling labels like 'A):' and fixing lines like I): albuminum humanum colloidale 0.5 mg, stanni(II) chloridum dihydricum 0.2 mg in oddb2xml. Changes look good.
Running jobs/import_swissmedic_only update_compositions 00613 now reports for the first time
Total time to update: 1.00 [m] Checked compositions: 7 Deleted compositions: 0 Updated agents: 0 New agents: 0
The link to http://oddb-ci2.dyndns.org/de/gcc/drug/reg/00613/seq/01 works fine. Running now twice jobs/import_swissmedic_only update_compositions to see whether the second time I have also 0 new agents.
But the import seems to get stuck at
2015-03-30 19:01:38 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:643 update_compositions delete_agent 0 atc_idx 0 :!registration,00654!sequence,01!composition,9485826!active_agent,Polysaccharida Neisseriae Meningitidis C. Polysaccharida Neisseriae Meningitidis C
Aborting it at 19:43. Now the CPU load goes up to 200% for over 2 minutes. Ag 19:49 kill -9 for the import.
Now http://oddb-ci2.dyndns.org/de/gcc/drug/reg/00654/seq/01 shows NoMethodError undefined method `resolve' for nil:NilClass, too. Running jobs/import_swissmedic_only update_compositions 00654 to try to catch the error.
Import fails with
Error: NoMethodError Message: undefined method `name' for nil:NilClass Backtrace: (eval):2:in `name' /var/www/oddb.org/src/util/oddbapp.rb:1207:in `block in search_single_substance' /var/www/oddb.org/src/util/oddbapp.rb:1207:in `each' /var/www/oddb.org/src/util/oddbapp.rb:1207:in `find' /var/www/oddb.org/src/util/oddbapp.rb:1207:in `search_single_substance' /var/www/oddb.org/src/util/oddbapp.rb:1337:in `substance' /var/www/oddb.org/src/plugin/swissmedic.rb:935:in `update_substance' /var/www/oddb.org/src/plugin/swissmedic.rb:573:in `update_active_agent' /var/www/oddb.org/src/plugin/swissmedic.rb:632:in `block (2 levels) in update_compositions' /var/www/oddb.org/src/plugin/swissmedic.rb:630:in `each' /var/www/oddb.org/src/plugin/swissmedic.rb:630:in `each_with_index' /var/www/oddb.org/src/plugin/swissmedic.rb:630:in `block in update_compositions' /var/www/oddb.org/src/plugin/swissmedic.rb:628:in `each' /var/www/oddb.org/src/plugin/swissmedic.rb:628:in `each_with_index' /var/www/oddb.org/src/plugin/swissmedic.rb:628:in `update_compositions' /var/www/oddb.org/src/plugin/swissmedic.rb:122:in `block in update' /usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.3.1/lib/rubyXL/worksheet.rb:23:in `block in each' /usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.3.1/lib/rubyXL/worksheet.rb:23:in `each' /usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.3.1/lib/rubyXL/worksheet.rb:23:in `each' /var/www/oddb.org/src/plugin/swissmedic.rb:104:in `update' /var/www/oddb.org/src/util/updater.rb:444:in `block in update_swissmedic' /var/www/oddb.org/src/util/updater.rb:549:in `call' /var/www/oddb.org/src/util/updater.rb:549:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:442:in `update_swissmedic' jobs/import_swissmedic_only:29:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_swissmedic_only:12:in `<module:Util>' jobs/import_swissmedic_only:11:in `<module:ODDB>' jobs/import_swissmedic_only:10:in `<main>'
Using bin/admin on oddb-ci2 I find 32 substances which are nil
ch.oddb> -> 32
Running on thinpower I find 31
ch.oddb> substances.find_all{ |x| x == nil}.size
-> 31
Therefore I looks as this kind of errors could occure on thinpower too. Fixing oddbapp.rb to avoid this error.
Restarted oddbd and import_swissmedic_only. Now it passes after 00654.
Pushed commit in oddb2xml Fix handling of composition labels for --calc