When importing the PI there are quite a few that are read again each day, e.g. Patinfo - DE - A. Vogel Johannisöl["38549"]. I once reported the following problem ["38549", "A. Vogel Johannisöl", "name_base in registration should match name_base in patinfo", :!patinfo,1983.]
From the log I see that we decided that its PI must be created before we see that it must be deactivated. Searching for 38549 returned the following info
2016-04-05 02:31:58 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1228:in `parse_textinfo': parse_textinfo 1228: must create textinfo for pi fr 38549 of ["38549"] 2016-04-05 02:31:58 +0200: /var/www/oddb.org/src/plugin/text_info.rb:324:in `update_patinfo_lang': update_patinfo_lang #<struct Struct::SwissmedicMetaInfo iksnr="38549", authNrs=["38549"], atcCode=nil, title="Huile de millepertuis A. Vogel (Herbe de la St-Jean)", authHolder="Bioforce AG", substances=nil, type="pi", lang="fr", informationUpdate="02.2011", refdata=nil, xml_file="/var/www/oddb.org/data/details/38549_pi_fr_Huile_de_millepertuis_A__Vogel__Herbe_de_la_St_Jean__problem.xml", same_content_as_xml_file=true> ["fr"] 2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:294:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang fr existing oid 1983 ["Huile de millepertuis A. Vogel (Herbe de la St-Jean)", "Qu\u2019est-ce que l\u2019huile de millepertuis A.Vogel et quand est-elle utilis\u00E9e?"] 2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:295:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang fr new ["Huile de millepertuis A. Vogel (Herbe de la St-Jean)", "Qu\u2019est-ce que l\u2019huile de millepertuis A.Vogel et quand est-elle utilis\u00E9e?"] 2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1228:in `parse_textinfo': parse_textinfo 1228: must create textinfo for pi de 38549 of ["38549"] 2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:324:in `update_patinfo_lang': update_patinfo_lang #<struct Struct::SwissmedicMetaInfo iksnr="38549", authNrs=["38549"], atcCode=nil, title="A. Vogel Johannis\u00F6l", authHolder="Bioforce AG", substances=nil, type="pi", lang="de", informationUpdate="02.2011", refdata=nil, xml_file="/var/www/oddb.org/data/details/38549_pi_de_A__Vogel_Johannis_l_problem.xml", same_content_as_xml_file=true> ["de"] 2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:294:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang de existing oid 1983 ["A. Vogel Johannis\u00F6l", "Was sollte dazu beachtet werden?"] 2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:295:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang de new ["A. Vogel Johannis\u00F6l", "Was ist A.Vogel Johannis\u00F6l und wann wird es angewendet?"] 2016-04-05 09:59:44 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:376:in `block in delete': (4 of 84) iksnr 38549 seqnr 01 pack 014 ODDB::Package 2016-04-05 11:11:10 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:351:in `block in deactivate': : deactivate iksnr '38549' seqnr 01 pack 10 2016-04-05 11:11:14 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:351:in `block in deactivate': : deactivate iksnr '38549' seqnr 00 pack 10 2016-04-05 12:01:55 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:377:in `block in delete': (4 of 84) iksnr 38549 seqnr 01 pack 014 ODDB::Package 2016-04-05 13:13:20 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:352:in `block in deactivate': : deactivate iksnr '38549' seqnr 01 pack 10 2016-04-05 13:13:25 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:352:in `block in deactivate': : deactivate iksnr '38549' seqnr 00 pack 10
Found the problem. Must change unit tests, too. Running jobs/update_textinfo_swissmedicinfo --target=pi to check whether this okay or not. It took less than 1 minute to read the packages into the memory. Importer uses 3100 MB of memory.
Seems to work better, as registration 38549 no longer appears. But must add some problematic names for
7680632350013_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override 7680632350020_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override 7680632350037_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override 7680632350044_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override 7680632350013_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override 7680632350020_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override 7680632350037_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override 7680632350044_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override 7680632350013_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override 7680632350020_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override 7680632350037_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override 7680632350044_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override 7680632350013_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override 7680632350020_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override 7680632350037_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override 7680632350044_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override
There are still some PI reimported. Found case of 40783 "Isoptin retard 120" which has 1 sequence 01, but no packages at all. Why?
The packages 016, 024 and 032 were present in March and were replaced by the packages 001, 002 and 003. (Probably the entries in March were just completly wrong!). Looking in the log I found an entry 2016-04-04 18:26:57 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:107:in `block in store_found_packages': store_found_packages: recreate_missing ["40783", "01", "001"]. But looks that this did not work correctly.
I think I overlooked the following lines in the update log
2016-04-05 13:15:48 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:824:in `block in update_compositions': delete_composition iksnr 40783 seqnr 01 comp_idx 0 comp.oid 9473753 comp.pointer :!registration,40783!sequence,01!composition,9473753. NoMethodError: undefined method `active_sequences' for Verapamili Hydrochloridum:ODDB::Substance when updating index 'substance_index_sequence' with a ODDB::Substance ["/var/www/oddb.org/src/util/language.rb:53:in `method_missing'", "(eval):2:in `block in proc_instance_target'", "/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:249:in `call'", "/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:249:in `update_origin'"] [...]
Adding a --check option to import_swissmedic_only to force validating that each package exists. THis check works now fine.
But after running import_swissmedic with the old packages, I found that for 65234/01/002 the package size is not updated correctly. Using bin/admin I found
ch.oddb> registration('65234').sequence('01').package('001').parts.size
-> 1
ch.oddb> registration('65234').sequence('01').package('002').parts.size
-> 0
Pushed commits:
Pulled onto thinpower. Restarted import_daily. Import_daily begins to eat memory now at 7015MB (14:56). 7544MB at 15.03. 7948MB at 15.04 while job post processing for FI is running. Going up to 10.7 while reading packages for PI. GOes up to 11.1GB job post processing for PI is running.
The parts are now created, but not the dose. Why? Looking via bin/admin
ch.oddb> registration('59322').sequence('05').compositions.first.galenic_form
-> Kapseln
ch.oddb> registration('59322').sequence('05').package('010').parts.first.comparable_size
-> 50
ch.oddb> registration('59322').sequence('05').package('12').parts.first.comparable_size
-> 0
Therefore somehow we never call part.size=.
Pushed commits:
Pulling and starting import_swissmedic on thinpower.
Adding a check to exit swissmedic import if more than 16 GB are used to avoid taking the server down. Log-Entries look now like this
2016-04-06 13:55:42 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:162:in `trace_memory_useage': Using 1489 MB of memory. Limit is 16384 and a failure produces an additional line like 2016-04-06 13:40:05 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:167:in `block in trace_memory_useage': Aborting as using 1489 MB of memory > than limit of 1024. Then a mail with the full stack trace of the main thread is sent, which should help searching for the error.
Maybe we should add this to all updaters? No. Pushed commit https://github.com/ngiger/oddb.org/commit/d971e192af93f5164bdf7a37fcf199159dc79a0e1Raise error if too much memory is used
Fixed the non appearing result for ebixa on ch.oddb.org via the following bin/admin command ch.oddb>registration('55828').sequence('03').package('888').sequence = registration('55828').sequence('03'); registration('55828').sequence('03').odba_store