view · edit · sidebar · attach · print · history

< Niklaus.20160411-textinfo-error | Index | Niklaus.20160405-changes-for-HIN >>

20160406-importonly-active-PI

Summary

  • Import only PI for active registrations
  • Search of ebixa does not work on ch.oddb.org]
  • Fix import swissmedic

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)
  • BSV-Plugin does not send mail. Is it sending two mails at once?
  • remove obsolete pdf_patinfo in model, admin/views

Import only PI for active registration

When importing the PI there are quite a few that are read again each day, e.g. Patinfo - DE - A. Vogel Johannisöl["38549"]. I once reported the following problem ["38549", "A. Vogel Johannisöl", "name_base in registration should match name_base in patinfo", :!patinfo,1983.]

From the log I see that we decided that its PI must be created before we see that it must be deactivated. Searching for 38549 returned the following info

2016-04-05 02:31:58 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1228:in `parse_textinfo': parse_textinfo 1228: must create textinfo for pi fr 38549 of ["38549"]
2016-04-05 02:31:58 +0200: /var/www/oddb.org/src/plugin/text_info.rb:324:in `update_patinfo_lang': update_patinfo_lang #<struct Struct::SwissmedicMetaInfo iksnr="38549", authNrs=["38549"], atcCode=nil, title="Huile de millepertuis A. Vogel (Herbe de la St-Jean)", authHolder="Bioforce AG", substances=nil, type="pi", lang="fr", informationUpdate="02.2011", refdata=nil, xml_file="/var/www/oddb.org/data/details/38549_pi_fr_Huile_de_millepertuis_A__Vogel__Herbe_de_la_St_Jean__problem.xml", same_content_as_xml_file=true> ["fr"]
2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:294:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang fr existing oid 1983 ["Huile de millepertuis A. Vogel (Herbe de la St-Jean)", "Qu\u2019est-ce que l\u2019huile de millepertuis A.Vogel et quand est-elle utilis\u00E9e?"]
2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:295:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang fr new ["Huile de millepertuis A. Vogel (Herbe de la St-Jean)", "Qu\u2019est-ce que l\u2019huile de millepertuis A.Vogel et quand est-elle utilis\u00E9e?"]
2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1228:in `parse_textinfo': parse_textinfo 1228: must create textinfo for pi de 38549 of ["38549"]
2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:324:in `update_patinfo_lang': update_patinfo_lang #<struct Struct::SwissmedicMetaInfo iksnr="38549", authNrs=["38549"], atcCode=nil, title="A. Vogel Johannis\u00F6l", authHolder="Bioforce AG", substances=nil, type="pi", lang="de", informationUpdate="02.2011", refdata=nil, xml_file="/var/www/oddb.org/data/details/38549_pi_de_A__Vogel_Johannis_l_problem.xml", same_content_as_xml_file=true> ["de"]
2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:294:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang de existing oid 1983 ["A. Vogel Johannis\u00F6l", "Was sollte dazu beachtet werden?"]
2016-04-05 02:31:59 +0200: /var/www/oddb.org/src/plugin/text_info.rb:295:in `store_patinfo_for_all_packages': store_patinfo update for reg.iksnr 38549 lang de new ["A. Vogel Johannis\u00F6l", "Was ist A.Vogel Johannis\u00F6l und wann wird es angewendet?"]
2016-04-05 09:59:44 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:376:in `block in delete': (4 of 84) iksnr 38549 seqnr 01 pack 014 ODDB::Package
2016-04-05 11:11:10 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:351:in `block in deactivate': : deactivate iksnr '38549' seqnr 01 pack 10
2016-04-05 11:11:14 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:351:in `block in deactivate': : deactivate iksnr '38549' seqnr 00 pack 10
2016-04-05 12:01:55 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:377:in `block in delete': (4 of 84) iksnr 38549 seqnr 01 pack 014 ODDB::Package
2016-04-05 13:13:20 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:352:in `block in deactivate': : deactivate iksnr '38549' seqnr 01 pack 10
2016-04-05 13:13:25 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:352:in `block in deactivate': : deactivate iksnr '38549' seqnr 00 pack 10

Found the problem. Must change unit tests, too. Running jobs/update_textinfo_swissmedicinfo --target=pi to check whether this okay or not. It took less than 1 minute to read the packages into the memory. Importer uses 3100 MB of memory.

Seems to work better, as registration 38549 no longer appears. But must add some problematic names for

7680632350013_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override
7680632350020_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override
7680632350037_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override
7680632350044_pi_fr: 'Paracétamol Spirig HC® 1000 mg' # != override
7680632350013_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override
7680632350020_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override
7680632350037_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override
7680632350044_pi_de: 'Paracetamol Spirig HC® 1000 mg' # != override
7680632350013_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override
7680632350020_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override
7680632350037_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override
7680632350044_pi_fr: 'Paracétamol Spirig HC® 500 mg' # != override
7680632350013_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override
7680632350020_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override
7680632350037_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override
7680632350044_pi_de: 'Paracetamol Spirig HC® 500 mg' # != override 

There are still some PI reimported. Found case of 40783 "Isoptin retard 120" which has 1 sequence 01, but no packages at all. Why? The packages 016, 024 and 032 were present in March and were replaced by the packages 001, 002 and 003. (Probably the entries in March were just completly wrong!). Looking in the log I found an entry 2016-04-04 18:26:57 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:107:in `block in store_found_packages': store_found_packages: recreate_missing ["40783", "01", "001"]. But looks that this did not work correctly.

I think I overlooked the following lines in the update log

2016-04-05 13:15:48 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:824:in `block in update_compositions': delete_composition iksnr 40783 seqnr 01 comp_idx 0 comp.oid 9473753 comp.pointer :!registration,40783!sequence,01!composition,9473753.
NoMethodError: undefined method `active_sequences' for Verapamili Hydrochloridum:ODDB::Substance when updating index 'substance_index_sequence' with a ODDB::Substance
["/var/www/oddb.org/src/util/language.rb:53:in `method_missing'", "(eval):2:in `block in proc_instance_target'", "/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:249:in `call'", "/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:249:in `update_origin'"]
[...]

Adding a --check option to import_swissmedic_only to force validating that each package exists. THis check works now fine.

But after running import_swissmedic with the old packages, I found that for 65234/01/002 the package size is not updated correctly. Using bin/admin I found

ch.oddb> registration('65234').sequence('01').package('001').parts.size
-> 1
ch.oddb> registration('65234').sequence('01').package('002').parts.size
-> 0

Pushed commits:

Pulled onto thinpower. Restarted import_daily. Import_daily begins to eat memory now at 7015MB (14:56). 7544MB at 15.03. 7948MB at 15.04 while job post processing for FI is running. Going up to 10.7 while reading packages for PI. GOes up to 11.1GB job post processing for PI is running.

The parts are now created, but not the dose. Why? Looking via bin/admin

ch.oddb> registration('59322').sequence('05').compositions.first.galenic_form
-> Kapseln
ch.oddb> registration('59322').sequence('05').package('010').parts.first.comparable_size  
-> 50   
ch.oddb> registration('59322').sequence('05').package('12').parts.first.comparable_size
-> 0 

Therefore somehow we never call part.size=.

Pushed commits:

Pulling and starting import_swissmedic on thinpower.

Out of Memory while running swissmedic import

Adding a check to exit swissmedic import if more than 16 GB are used to avoid taking the server down. Log-Entries look now like this 2016-04-06 13:55:42 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:162:in `trace_memory_useage': Using 1489 MB of memory. Limit is 16384 and a failure produces an additional line like 2016-04-06 13:40:05 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb:167:in `block in trace_memory_useage': Aborting as using 1489 MB of memory > than limit of 1024. Then a mail with the full stack trace of the main thread is sent, which should help searching for the error.

Maybe we should add this to all updaters? No. Pushed commit https://github.com/ngiger/oddb.org/commit/d971e192af93f5164bdf7a37fcf199159dc79a0e1Raise error if too much memory is used

Search of ebixa does not work on ch.oddb.org

Fixed the non appearing result for ebixa on ch.oddb.org via the following bin/admin command ch.oddb>registration('55828').sequence('03').package('888').sequence = registration('55828').sequence('03'); registration('55828').sequence('03').odba_store

view · edit · sidebar · attach · print · history
Page last modified on April 06, 2016, at 05:25 PM