view · edit · sidebar · attach · print · history

Index>

20150313-oddb2xml-all-compontents

Summary

  • Add all components in oddb2xml whether they appear in Wirkstoffe or not
  • import of epha_interactions reports 20002 items instead of 100001

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml

import of epha_interactions reports 20002 items instead of 100001

Got the following epha-import message on oddb-ci2 and thinpower

EphaInteractionPlugin.update latest /var/www/oddb.org/data/csv/interactions_de_utf8-latest.csv true via /var/www/oddb.org/data/csv/interactions_de_utf8.csv from https://download.epha.ch/data/matrix/matrix.csv
Added 20002 interactions from /var/www/oddb.org/data/csv/interactions_de_utf8-latest.csv

Does the epha-interaction import report a wrong size or does it really reimport everything twice? On thinpower bin/admin reports

 bin/admin
ch.oddb> epha_interactions.size
-> 0
ch.oddb> epha_interactions_hash.size
-> 10001
ch.oddb> 
ch.oddb> epha_interactions_hash.first
-> [#<ODBA::Stub:138337120#33357824 @odba_class=Array @odba_container=133393900#33345444>, #<ODBA::Stub:138336780#33347814 @odba_class=ODDB::EphaInteraction @odba_container=133393900#33345444>]
ch.oddb> epha_interactions_hash.first[0]
-> ["J05AE03", "P01BB51"]
ch.oddb> epha_interactions_hash.first[1]
-> J05AE03;Ritonavir;P01BB51;Proguanil, Kombinationen;Reduzierte Atovaquon/Proguanil-Konzentrationen;Ritonavir induziert CYP2C19  und kann damit zu einem verstärkten Abbau von Proguanil zu Cycloguanil (e
ch.oddb> 

Therefore it looks as if the import left a correct database.

On oddb-ci2 I found the following runtime error

Defaulting to UTF-8, set your own if this is incorrect.
D, [2015-03-11T10:34:40.073913 #27628] DEBUG -- : HTTPI executes HTTP GET using the httpclient adapter
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:589: [BUG] Segmentation fault
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0024 p:0055 s:0083 b:0081 l:000067 d:000080 BLOCK  /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:589
c:0023 p:---- s:0077 b:0077 l:000076 d:000076 FINISH
c:0022 p:---- s:0075 b:0075 l:000070 d:000074 IFUNC 
c:0021 p:---- s:0073 b:0073 l:000072 d:000072 CFUNC  :each
c:0020 p:---- s:0071 b:0071 l:000070 d:000070 CFUNC  :any?
c:0019 p:0040 s:0068 b:0068 l:000067 d:000067 METHOD /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:587
c:0018 p:0051 s:0064 b:0064 l:000063 d:000063 METHOD /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:37
c:0017 p:0051 s:0057 b:0057 l:000046 d:000056 BLOCK  /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:99
c:0016 p:---- s:0052 b:0052 l:000051 d:000051 FINISH
c:0015 p:---- s:0050 b:0050 l:000049 d:000049 CFUNC  :delete_if
c:0014 p:0073 s:0047 b:0047 l:000046 d:000046 METHOD /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:96
c:0013 p:0049 s:0042 b:0042 l:000028 d:000041 BLOCK  /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:105
c:0012 p:---- s:0039 b:0039 l:000038 d:000038 FINISH
c:0011 p:---- s:0037 b:0037 l:000036 d:000036 CFUNC  :each_value
c:0010 p:0011 s:0034 b:0034 l:000028 d:000033 BLOCK  /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:102
c:0009 p:0019 s:0032 b:0032 l:000031 d:000031 METHOD <internal:prelude>:10
c:0008 p:0050 s:0029 b:0029 l:000028 d:000028 METHOD /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:101
c:0007 p:0139 s:0021 b:0021 l:000020 d:000020 METHOD /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:81
c:0006 p:0030 s:0014 b:0014 l:000b38 d:000013 BLOCK  /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:490
c:0005 p:---- s:0011 b:0011 l:000010 d:000010 FINISH
c:0004 p:---- s:0009 b:0009 l:000008 d:000008 CFUNC  :loop
c:0003 p:0040 s:0006 b:0006 l:000b38 d:000005 BLOCK  /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:487
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP   

-- Ruby level backtrace information ----------------------------------------
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:487:in `block in start_cleaner'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:487:in `loop'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:490:in `block (2 levels) in start_cleaner'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:81:in `clean'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:101:in `_clean'
<internal:prelude>:10:in `synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:102:in `block in _clean'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:102:in `each_value'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:105:in `block (2 levels) in _clean'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:96:in `odba_retire'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:96:in `delete_if'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:99:in `block in odba_retire'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:37:in `object_id2ref'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:587:in `odba_unsaved?'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:587:in `any?'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:587:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:589:in `block in odba_unsaved?'

-- C level backtrace information -------------------------------------------
Oddb (OddbApp)() [0x5216d8]
Oddb (OddbApp)() [0x5694c9]
Oddb (OddbApp)(rb_bug+0xb4) [0x56a6e4]
Oddb (OddbApp)() [0x4b4e12]
/lib64/libpthread.so.0(+0x113d0) [0x7f4fb15193d0]
Oddb (OddbApp)(st_lookup+0xe) [0x4bce7e]
Oddb (OddbApp)() [0x50f12f]
Oddb (OddbApp)() [0x517a90]
Oddb (OddbApp)() [0x518333]
Oddb (OddbApp)(rb_yield_values2+0x34) [0x51e754]
Oddb (OddbApp)() [0x562dde]
Oddb (OddbApp)() [0x51061e]
Oddb (OddbApp)(rb_yield+0x47) [0x51e7a7]
Oddb (OddbApp)() [0x426577]
Oddb (OddbApp)() [0x426022]
Oddb (OddbApp)(st_foreach+0xbc) [0x4bdfec]
Oddb (OddbApp)() [0x426968]
Oddb (OddbApp)(rb_ensure+0xab) [0x41b14b]
Oddb (OddbApp)() [0x428c77]
Oddb (OddbApp)() [0x519637]
Oddb (OddbApp)() [0x51cb40]
Oddb (OddbApp)(rb_block_call+0x24) [0x51cce4]
Oddb (OddbApp)() [0x56241d]
Oddb (OddbApp)() [0x51dd51]
Oddb (OddbApp)() [0x511f3b]
Oddb (OddbApp)() [0x518333]
Oddb (OddbApp)(rb_yield_values+0xda) [0x51e8ba]
Oddb (OddbApp)() [0x4265ac]
Oddb (OddbApp)() [0x426022]
Oddb (OddbApp)(st_foreach+0xbc) [0x4bdfec]
Oddb (OddbApp)() [0x426968]
Oddb (OddbApp)(rb_ensure+0xab) [0x41b14b]
Oddb (OddbApp)(rb_hash_delete_if+0x4b) [0x428f8b]
Oddb (OddbApp)() [0x51dd51]
Oddb (OddbApp)() [0x511f3b]
Oddb (OddbApp)() [0x518333]
Oddb (OddbApp)(rb_yield+0x47) [0x51e7a7]
Oddb (OddbApp)() [0x425d02]
Oddb (OddbApp)() [0x426022]
Oddb (OddbApp)(st_foreach+0xbc) [0x4bdfec]
Oddb (OddbApp)() [0x426968]
Oddb (OddbApp)(rb_ensure+0xab) [0x41b14b]
Oddb (OddbApp)() [0x428d77]
Oddb (OddbApp)() [0x51dd51]
Oddb (OddbApp)() [0x511f3b]
Oddb (OddbApp)() [0x518333]
Oddb (OddbApp)() [0x51e5e4]
Oddb (OddbApp)(rb_rescue2+0x163) [0x41ae03]
Oddb (OddbApp)() [0x50da29]
Oddb (OddbApp)() [0x51dd51]
Oddb (OddbApp)() [0x511f3b]
Oddb (OddbApp)() [0x518333]
Oddb (OddbApp)() [0x5192c1]
Oddb (OddbApp)() [0x52777e]
Oddb (OddbApp)() [0x5278c9]
/lib64/libpthread.so.0(+0x91a7) [0x7f4fb15111a7]
/lib64/libc.so.6(clone+0x6d) [0x7f4fb090920d]

-- Other runtime information -----------------------------------------------
<..>

In oddbapp.rb replaced in method delete_all_epha_interactions odba_isolated_store by odba_store and added some more debug info in the import. Still the same problem!

Zeno also remarked that we should fulfill the following requirements

  • No mail if no import is done

We decided to implement (for epha_interactions) a utility procedure get_latest_file (there are already 5 procedure with the same name), namely

  • src/plugin/analysis.rb: def get_latest_file(lang = 'de')
  • src/plugin/swissmedic.rb: def get_latest_file(agent, keyword='Packungen', extension = '.xlsx')
  • src/plugin/medreg_pharmacy.rb: def get_latest_file
  • src/plugin/vaccines.rb: def get_latest_file
  • src/plugin/medreg_doctor.rb: def get_latest_file

This utility prodedure should be used later on for all occurences, see above. Found nr of occurrences of latest in the following files

src/plugin/analysis.rb:9
src/plugin/atc_less.rb:2
src/plugin/bsv_xml.rb:11
src/plugin/epha_interactions.rb:7
src/plugin/medreg_doctor.rb:12
src/plugin/medreg_pharmacy.rb:15
src/plugin/ouwerkerk.rb:2
src/plugin/swissmedic.rb:33
src/plugin/swissmedicjournal.rb:3
src/plugin/swissreg.rb:1
src/plugin/text_info.rb:8
src/plugin/vaccines.rb:10

Running the import skips now sending a message, when nothing is todo. Forcing an import by removing all concerned csv files (data/csv/interactions_de_utf8-2015.03.13.csv and data/csv/interactions_de_utf8-latest.csv).

Pushed commit Fix latest in epha_interactions.rb. Fix Reporting.

Add all components in oddb2xml whether they appear in Wirkstoffe or not

Must resolve the problem on how to avoid adding the the last part, eg. aqua ad 100 ml or Excipiens Ad Emulsionem Pro as a separate component. Also my local version mangles names like Globulina Equina (immunisé Avec Coeur, Tissu Pulmonaire, Reins De Porcins) to Reins De Porcins). Handling colors like "E 127" or conserv.: E 216, E 218 is also wrong. Current diff is Attach:oddb_xml_components.txt.

Pushed the commits

view · edit · sidebar · attach · print · history
Page last modified on March 13, 2015, at 05:51 PM