view · edit · sidebar · attach · print · history

Index>

20150701-refdatabase-oddb-org

Summary

  • Error loading oddb2xml
  • Adapt to changed URI of refdata in oddb2xml

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Display new fields (LABEL, MORE_INFO, CORRESP) for compositions in oddb.org.
  • Use refdatabase for oddb.org, too.

Invalid version of rubyntlm

Must fix the following error 

/usr/lib64/ruby/gems/2.1.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:53:
warning: already initialized constant Net::NTLM::VERSION::MAJOR
/usr/lib64/ruby/gems/2.1.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:5:
warning: previous definition of MAJOR was here
/usr/lib64/ruby/gems/2.1.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:54:
warning: already initialized constant Net::NTLM::VERSION::MINOR
/usr/lib64/ruby/gems/2.1.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:6:
warning: previous definition of MINOR was here
/usr/lib64/ruby/gems/2.1.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:55:
warning: already initialized constant Net::NTLM::VERSION::TINY
/usr/lib64/ruby/gems/2.1.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:7:
warning: previous definition of TINY was here
/usr/lib64/ruby/gems/2.1.0/gems/ntlm-http-0.1.1/lib/net/ntlm.rb:56:
warning: already initialized constant Net::NTLM::VERSION::STRING
/usr/lib64/ruby/gems/2.1.0/gems/rubyntlm-0.5.1/lib/net/ntlm/version.rb:8:
warning: previous definition of STRING was here
/usr/lib64/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:58:in 
`check_net_ntlm_version!': Invalid version of rubyntlm. Please use
v0.3.2+. (ArgumentError)
        from /usr/lib64/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb:19:in `initialize'
        from /usr/lib64/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi.rb:185:in `new'
        from /usr/lib64/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi.rb:185:in `load_adapter'
        from /usr/lib64/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi.rb:156:in `request'
        from /usr/lib64/ruby/gems/2.1.0/gems/httpi-2.4.1/lib/httpi.rb:127:in `get'
        from /usr/lib64/ruby/gems/2.1.0/gems/wasabi-3.5.0/lib/wasabi/resolver.rb:43:in `load_from_remote'
        from /usr/lib64/ruby/gems/2.1.0/gems/wasabi-3.5.0/lib/wasabi/resolver.rb:33:in `resolve'
        from /usr/lib64/ruby/gems/2.1.0/gems/wasabi-3.5.0/lib/wasabi/document.rb:142:in `xml'
        from /usr/lib64/ruby/gems/2.1.0/gems/wasabi-3.5.0/lib/wasabi/document.rb:160:in `parse'
        from /usr/lib64/ruby/gems/2.1.0/gems/wasabi-3.5.0/lib/wasabi/document.rb:147:in `parser'
        from /usr/lib64/ruby/gems/2.1.0/gems/wasabi-3.5.0/lib/wasabi/document.rb:64:in `soap_actions'
        from /usr/lib64/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/operation.rb:22:in `ensure_exists!'
        from /usr/lib64/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/operation.rb:15:in `create'
        from /usr/lib64/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/client.rb:32:in `operation'
        from /usr/lib64/ruby/gems/2.1.0/gems/savon-2.11.1/lib/savon/client.rb:36:in `call'
        from /usr/lib64/ruby/gems/2.1.0/gems/oddb2xml-2.2.0/lib/oddb2xml/downloader.rb:241:in `download'
        from /usr/lib64/ruby/gems/2.1.0/gems/oddb2xml-2.2.0/lib/oddb2xml/cli.rb:280:in `download'
        from /usr/lib64/ruby/gems/2.1.0/gems/oddb2xml-2.2.0/lib/oddb2xml/cli.rb:70:in `block in run'
        from /usr/lib64/ruby/gems/2.1.0/gems/oddb2xml-2.2.0/lib/oddb2xml/cli.rb:69:in `each'
        from /usr/lib64/ruby/gems/2.1.0/gems/oddb2xml-2.2.0/lib/oddb2xml/cli.rb:69:in `run'
        from /usr/lib64/ruby/gems/2.1.0/gems/oddb2xml-2.2.0/bin/oddb2xml:49:in `<top (required)>'
        from /usr/bin/oddb2xml:23:in `load'
        from /usr/bin/oddb2xml:23:in `<main>'

This was fixed with commit Bump version to 2.2.1. Fix uninitialized constant Net::NTLM::VERSION

The problem lies that savon and mechanize each use (by direct or indirect require) ruby-ntlm. Depending whether we used bundle exec bin/oddb2xml -e or an installed version of oddb2xml 2.2.0 via oddb2xml -e the order of includes seems to differ and used different version of rubyntlm. Seems to be the issue reported at https://github.com/WinRb/rubyntlm/issues/13

Ruby Mechanize, Savon issues are reported here

 https://github.com/sparklemotion/mechanize/issues/436
 https://github.com/savonrb/savon/issues/703

Adapt to changed URI of refdata in oddb.org

Must port adaptations of oddb2xml to oddb.org.

The following jobs must pass without problems

  • jobs/update_atc_less (atc_less in src/util/update.rb
  • jobs/update/bsv_only (update_bsv in src/util/update.rb via BsvXmlPlugin in src/plugin/bsv_xml)

Therefore focussing on making test/test_util/updater.rb pass again. Done.

test/test_plugin/bsv_xml.rb making it use VCR. But first copied test/test_plugin/swissindex.rb to test/test_plugin/refdata.rb and making it pass again.

Renamed jobs/update_swissindex_pharma to jobs/update_refdata. Letting run jobs/update_refdata on oddb-ci2 to check the results. Failed because I have an error in the started services 

Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: Refdata - 07/2015 size 1420 with 0 attachments. Plugin: ODDB::RefdataPlugin
Error: DRb::DRbConnError
Message: druby://localhost:50001 - #<Errno::ECONNREFUSED: Connection refused - connect(2)>
Backtrace:

Till now port 50001 was used by SwissindexPharma and port 50002 by SwissindexPharma (which handled also calls for migel).

Must delete old services

  • swissindex_pharma (port 50001)
  • swissindex_nonpharma (port 50002)

and add

  • refdata (port 50001) (run contains exec sudo -u apache /usr/local/bin/ruby ext/refdata/bin/refdatad)
  • swissindex (Only migel, port 50002), run ext/refdata/bin/swissindexd

Services start okay now. But import failed because of a spelling mistake. Fixing another problem with a wrong number of arguments which was not catched by an appropriate unit test.

To check whether the import is okay, I must look at pages like http://oddb-ci2.dyndns.org/de/gcc/drug/reg/65154/seq/01/pack/001 to see whether the pharmacode must be 5950031 and not 0.

Waiting for jobs/update_refdata to finish. Did something went wrong, as I have way many outdated packages 

Checked 23744 packages
Updated in trade     (out_of_trade:false): 0 packages
Updated out of trade (out_of_trade:true) : 2100 packages
Updated pharmacode: 0 packages
Deleted pharmacode: 4895 packages

Updated in trade     (out_of_trade:false): 0 packages
Check swissindex by eancode and then check if the package is out of trade (true) in ch.oddb,
if so the package becomes in trade (false)

Updated out of trade (out_of_trade:true) : 2100 packages
If there is no eancode in swissindex and the package is in trade in ch.oddb,
then the package becomes out of trade (true) in ch.oddb
7680614540036: http://ch.oddb.apache.org/de/gcc/drug/reg/61454/seq/02/pack/003
7680614540043: http://ch.oddb.apache.org/de/gcc/drug/reg/61454/seq/02/pack/004
7680614540050: http://ch.oddb.apache.org/de/gcc/drug/reg/61454/seq/03/pack/005
<..>
7680656570022: http://ch.oddb.apache.org/de/gcc/drug/reg/65657/seq/02/pack/002

Updated pharmacode: 0 packages
If the package does not have a pharmacode and there is a pharmacode found in swissindex,
then put the pharmacode into ch.oddb

Deleted pharmacode: 4895 packages
If there is no eancode in swissindex then delete the according pharmacode in ch.oddb
7680002820016: http://ch.oddb.apache.org/de/gcc/drug/reg/00282/seq/01/pack/001
7680002880010: http://ch.oddb.apache.org/de/gcc/drug/reg/00288/seq/01/pack/001
<..>

Checking what is going on with Levetiracetam Desitin GTIN: 7680620690084 and http://ch.oddb.org/de/gcc/drug/reg/62069/seq/01/pack/008

Checking in bin/admin works like this 

h.oddb> $res = $plugin.search_item(7680620690084)
-> Hash
ch.oddb> $res.keys
-> [:atype, :gtin, :phar, :swmc_authnr, :name_de, :name_fr, :atc, :auth_holder_name, :auth_holder_gln, :@dt]
ch.oddb> $res.values[0..5]
-> ["PHARMA", "7680620690084", "5819012", "62069008", "LEVETIRACETAM DESITIN Mini Filmtab 250 mg 30 Stk", "LEVETIRACETAM DESITIN mini cpr pel 250 mg 30 pce"]
ch.oddb> $res.values[6..10]
-> ["N03AX14", "Desitin Pharma GmbH", "7601001320451", "2015-07-01T00:00:00"]


ch.oddb> $res = $plugin.search_item(7680405500010)
-> Hash
ch.oddb> $res.keys
-> [:atype, :gtin, :phar, :swmc_authnr, :name_de, :name_fr, :atc, :auth_holder_name, :auth_holder_gln, :@dt]
ch.oddb> $res.values[0..5]
-> ["PHARMA", "7680405500010", "6131772", "40550001", "HIRUDOID forte Gel 4.45 mg/g Tb 40 g", "HIRUDOID forte gel 4.45 mg/g tb 40 g"]
ch.oddb> $res.values[6..10]
-> ["C05BA01", "Medinova AG", "7601001002258", "2015-07-01T00:00:00"]

ch.oddb> $res = $plugin.check_item(7680405500010)
-> 
ch.oddb> $res.class
-

Looks like search_item works, but not check_item. Must investigate. Has no spec test. Found the culprit. Search in the NonPharma and not in the Pharma-elements.

Rerunning twice the import resulted in a second mail like 

Checked 23744 packages
Updated in trade     (out_of_trade:false): 0 packages
Updated out of trade (out_of_trade:true) : 1 packages
Updated pharmacode: 1248 packages
Deleted pharmacode: 0 packages

Updated in trade     (out_of_trade:false): 0 packages
Check swissindex by eancode and then check if the package is out of trade (true) in ch.oddb,
if so the package becomes in trade (false)

Updated out of trade (out_of_trade:true) : 1 packages
If there is no eancode in swissindex and the package is in trade in ch.oddb,
then the package becomes out of trade (true) in ch.oddb
7680000000007: http://ch.oddb.apache.org/de/gcc/drug/reg/00000/seq/00/pack/000

Updated pharmacode: 1248 packages
If the package does not have a pharmacode and there is a pharmacode found in swissindex,
then put the pharmacode into ch.oddb
7680002820016: http://ch.oddb.apache.org/de/gcc/drug/reg/00282/seq/01/pack/001
<...>
7680579740021: http://ch.oddb.apache.org/de/gcc/drug/reg/57974/seq/01/pack/002

Deleted pharmacode: 0 packages
If there is no eancode in swissindex then delete the according pharmacode in ch.oddb

Pushed commit Changed URL for refdata

view · edit · sidebar · attach · print · history
Page last modified on July 02, 2015, at 08:32 AM