view · edit · sidebar · attach · print · history

Index>

20150703-new-packungen-xlsx

Summary

  • Adapt ch.oddb.org to changed format of Packungen-xlsx
  • Fix mess when loading ruby-ntlm

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Display new fields (LABEL, MORE_INFO, CORRESP) for compositions in oddb.org.
  • Use refdatabase for oddb.org, too.

Adapt ch.oddb.org to changed format of Packungen-xlsx

Swissmedic changed format of Packungen.xlsx which broke havoc on ch.oddb.org. Zeno reverts Postgres database to state of July 1.

Changes in format are:

  • column "Heilmittelcode" from G to E
  • Added new columns O "Abgabekategorie Dosisstärke" and P "Abgabekategorie Präparat"
  • Added new columns U "Gentechnisch hergestellte Wirkstoffe", V "Kategorie bei Insulinen", W "Verz. bei betäubunsmittel-haltigen Präparaten"
  • column heading are now in german and french

Must fix the error in gem swissmedic-diff. Added a test for comparing (a small subset) of columns of the Packungen.xlsx of last june and july. When running with RubyXL-3.3.8 I get the following error 

 1) Error:
ODDB::SwissmedicPluginTest#test_diff_new_format_july_2015:
NoMethodError: undefined method `size' for nil:NilClass
    /opt/src/swissmedic-diff/lib/swissmedic-diff.rb:301:in `block in each_valid_row'
    /home/niklaus/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/rubyXL-3.3.8/lib/rubyXL/worksheet.rb:23:in `block in each'
    /home/niklaus/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/rubyXL-3.3.8/lib/rubyXL/worksheet.rb:23:in `each'
    /home/niklaus/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/rubyXL-3.3.8/lib/rubyXL/worksheet.rb:23:in `each'
    /opt/src/swissmedic-diff/lib/swissmedic-diff.rb:297:in `each_valid_row'
    /opt/src/swissmedic-diff/lib/swissmedic-diff.rb:184:in `_known_data'
    /opt/src/swissmedic-diff/lib/swissmedic-diff.rb:177:in `known_data'
    /opt/src/swissmedic-diff/lib/swissmedic-diff.rb:96:in `diff'
    test/test_swissmedic-diff.rb:34:in `test_diff_new_format_july_2015'

But on thinpower I have rubyXL 3.3.1 installed. Get the same error with both. Therefore I wonder why I did not get the same error yesterday on thinpower. I get the same error running a similar test with oddb.org/src/plugin/swissmedic.rb.

It gets tricky to compare two rows if the columns do not match together.

Removed support for old XLS-files. Reworked unit tests for comparing two XLSX-files. Added explicit test when a column is deleted and raise an error in this case.

All headernames must now match the pattern for the supported formats.

Pushed commit Bumped version to 0.2.0. Remove support for old *.xls. Add support for new SwissMedic-XLSX

Fix mess when loading ruby-ntlm

Tracking why ext/bin/refdatad generates a Invalid version of rubyntlm. Please use v0.3.2+.. Created a minimal Gemfile to run the ext/refdata/test/refdata.rb as 

source "http://rubygems.org"
if /^2/.match(RUBY_VERSION)
  ruby '2.1.2'
  gem 'dbi', :git => 'https://github.com/ngiger/ruby-dbi'
  gem 'syck'
else
  ruby "1.9.3"
  gem 'dbi', :git => 'https://github.com/ngiger/ruby-dbi'
end

  gem 'mechanize', '2.5.1'
  gem 'minitest'
  gem 'savon'
  gem 'rclconf'
  gem 'flexmock'
  gem 'vcr'
  gem 'webmock'
  gem 'multi_json', '1.8.4'
  gem "rubyzip", ">= 1.1.6" , :require => 'zip'

which results in the Gemfile.lock 

GIT
  remote: https://github.com/ngiger/ruby-dbi
  revision: 3b9421fb94f43438962ebc998d36dd94e1881ded
  specs:
    dbi (0.4.6)
      deprecated (= 2.0.1)

GEM
  remote: http://rubygems.org/
  specs:
    addressable (2.3.8)
    akami (1.3.1)
      gyoku (>= 0.4.0)
      nokogiri
    builder (3.2.2)
    crack (0.4.2)
      safe_yaml (~> 1.0.0)
    deprecated (2.0.1)
    domain_name (0.5.24)
      unf (>= 0.0.5, < 1.0.0)
    flexmock (1.3.3)
    gyoku (1.3.1)
      builder (>= 2.1.2)
    httpi (2.4.1)
      rack
    mechanize (2.5.1)
      domain_name (~> 0.5, >= 0.5.1)
      mime-types (~> 1.17, >= 1.17.2)
      net-http-digest_auth (~> 1.1, >= 1.1.1)
      net-http-persistent (~> 2.5, >= 2.5.2)
      nokogiri (~> 1.4)
      ntlm-http (~> 0.1, >= 0.1.1)
      webrobots (~> 0.0, >= 0.0.9)
    mime-types (1.25.1)
    mini_portile (0.6.2)
    minitest (5.7.0)
    multi_json (1.8.4)
    net-http-digest_auth (1.4)
    net-http-persistent (2.9.4)
    nokogiri (1.6.6.2)
      mini_portile (~> 0.6.0)
    nori (2.6.0)
    ntlm-http (0.1.1)
    rack (1.6.4)
    rclconf (1.0.0)
    rubyzip (1.1.7)
    safe_yaml (1.0.4)
    savon (2.11.1)
      akami (~> 1.2)
      builder (>= 2.1.2)
      gyoku (~> 1.2)
      httpi (~> 2.3)
      nokogiri (>= 1.4.0)
      nori (~> 2.4)
      wasabi (~> 3.4)
    unf (0.1.4)
      unf_ext
    unf_ext (0.0.7.1)
    vcr (2.9.3)
    wasabi (3.5.0)
      httpi (~> 2.0)
      nokogiri (>= 1.4.2)
    webmock (1.21.0)
      addressable (>= 2.3.6)
      crack (>= 0.3.2)
    webrobots (0.1.1)

PLATFORMS
  ruby

DEPENDENCIES
  dbi!
  flexmock
  mechanize (= 2.5.1)
  minitest
  multi_json (= 1.8.4)
  rclconf
  rubyzip (>= 1.1.6)
  savon
  vcr
  webmock

BUNDLED WITH
   1.10.4

Now I see the error RefdataArticle.search_item(7680620690084) failed: Invalid version of rubyntlm. Please use v0.3.2+.. Modifying the code to raise an ArgumentError when catching this error, was we are unable to rescue from it. Now the Unit-Tests fail very fast.

Error is generate in line 54 of file /home/niklaus/.rbenv/versions/1.9.3-p551/lib/ruby/gems/1.9.1/gems/httpi-2.4.1/lib/httpi/adapter/net_http.rb

Now I get very quick an error when starting ext/refdata/bin/refdatad, as seen here 

/opt/src/oddb.org/ext/refdata/src/refdata.rb:21:in `check_net_ntlm_version': Invalid version of rubyntlm. Please use v0.3.2+. (ArgumentError)
	from /opt/src/oddb.org/ext/refdata/src/refdata.rb:26:in `<module:Refdata>'
	from /opt/src/oddb.org/ext/refdata/src/refdata.rb:11:in `<module:ODDB>'
	from /opt/src/oddb.org/ext/refdata/src/refdata.rb:10:in `<top (required)>'
	from /home/niklaus/.rbenv/versions/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
	from /home/niklaus/.rbenv/versions/1.9.3-p551/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
	from ext/refdata/bin/refdatad:7:in `<main>'

Adding gem 'rubyntlm', '>= 0.3.2' to the Gemfile make the test pass again. Pushed commit Fail fast if wrong version rubyntlm

Installing gem with identical version on oddb-ci as on thinpower to find the culprit. Had to push commit Force loading of rubyntlm early to avoid error to fix starting bin/oddbd and ext/refdata/bin/refdatad.

In bin/oddbd output I see the following error 

undefined method `[]' for #<DRb::DRbUnknown:0x000000095ee440>
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/session.rb:208:in `import_cookies'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/session.rb:360:in `process'
/var/www/oddb.org/src/util/session.rb:125:in `process'
/usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/session.rb:172:in `block in drb_process'
<internal:prelude>:10:in `synchronize'

C

view · edit · sidebar · attach · print · history
Page last modified on July 03, 2015, at 11:00 PM