view · edit · sidebar · attach · print · history

20131014-swissmedic-update

<< | Index | >>


Summary

  • Swissmedic update must complete

Commits

Index

---

Swissmedic update must complete

We have seen the following error when running the swissmedic-update with the new Packungen.xls

Plugin: ODDB::SwissmedicPlugin
Error: NoMethodError
Message: undefined method `round' for #<String:0x007fae12581cc8>
Backtrace:
/usr/local/lib/ruby/gems/1.9.1/gems/spreadsheet-0.6.5.9/lib/spreadsheet/excel/row.rb:57:in
`_datetime'
/usr/local/lib/ruby/gems/1.9.1/gems/spreadsheet-0.6.5.9/lib/spreadsheet/excel/row.rb:48:in
`_date'
/usr/local/lib/ruby/gems/1.9.1/gems/spreadsheet-0.6.5.9/lib/spreadsheet/excel/row.rb:16:in
`date'
/var/www/oddb.org/src/plugin/swissmedic.rb:113:in `date_cell'
/var/www/oddb.org/src/plugin/swissmedic.rb:729:in `update_registration'
/var/www/oddb.org/src/plugin/swissmedic.rb:779:in `block in
update_registrations'
/var/www/oddb.org/src/plugin/swissmedic.rb:778:in `each'
/var/www/oddb.org/src/plugin/swissmedic.rb:778:in `update_registrations'
/var/www/oddb.org/src/plugin/swissmedic.rb:48:in `update'
/var/www/oddb.org/src/util/updater.rb:387:in `block in update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:488:in `call'
/var/www/oddb.org/src/util/updater.rb:488:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:385:in `update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:195:in `run'
/var/www/oddb.org/jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
/var/www/oddb.org/jobs/import_daily:12:in `<module:Util>'
/var/www/oddb.org/jobs/import_daily:11:in `<module:ODDB>'
/var/www/oddb.org/jobs/import_daily:10:in `<main>

Looking at the STDOUT of the import run I see sometimes errors like

[8537.0, 1.0, "Aspirin, Tabletten", "Bayer (Schweiz) AG", "01.01.1.", "N02BA01", "Synthetika human", 13331.0, 13331.0, 42864.0, 11.0, "20", "Tablette(n)", "D", "acidum acetylsalicylicum", "acidum acetylsalicylicum 500 mg, excipiens pro compresso.", "Analgetikum, Antipyretikum", nil, 0]> date 9
SBSM::InvalidDataError: e_invalid_ean_code when updating index 'oddb_package_name_with_size_company_name_ean13_fi' with a ODDB::Package
["/var/www/oddb.org/src/model/ean13.rb:12:in `initialize'", "/var/www/oddb.org/src/model/ean13.rb:26:in `new'", "/var/www/oddb.org/src/model/ean13.rb:26:in `new_unchecked'", "/var/www/oddb.org/src/model/package.rb:163:in `barcode'"]
[...]

This seems to happen several times, but nothing harmfull noted.

The import stops when analysing the line [12548.0, 1.0, "Dul-X Classic, Emulsion", "Melisana AG", "07.10.4.", "M02AX10", "Phytotherapeutika", 16771.0, 16771.0, 42199.0, 60.0, "125", "ml", "D", "levomentholum, anisi stellati aetheroleum, eucalypti aetheroleum, gaultheriae aetheroleum, citronellae aetheroleum, menthae piperitae aetheroleum, rosmarini aetheroleum, arnicae floris extractum oleosum", nil, "Einreibemittel bei Muskelschmerzen", nil, 0]

But the callstack on my VM is different and looks like this:

Plugin: ODDB::SwissmedicPlugin
Error: NoMethodError
Message: undefined method `gsub' for nil:NilClass
Backtrace:
/var/www/oddb.org/src/plugin/swissmedic.rb:548:in `update_compositions'
/var/www/oddb.org/src/plugin/swissmedic.rb:784:in `block in update_registrations'
/var/www/oddb.org/src/plugin/swissmedic.rb:780:in `each'
/var/www/oddb.org/src/plugin/swissmedic.rb:780:in `update_registrations'
/var/www/oddb.org/src/plugin/swissmedic.rb:48:in `update'
/var/www/oddb.org/src/util/updater.rb:387:in `block in update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:488:in `call'
/var/www/oddb.org/src/util/updater.rb:488:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:385:in `update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:195:in `run'
/var/www/oddb.org/jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
/var/www/oddb.org/jobs/import_daily:12:in `<module:Util>'
/var/www/oddb.org/jobs/import_daily:11:in `<module:ODDB>'
/var/www/oddb.org/jobs/import_daily:10:in `<main>'

Spreadsheet-gem has version 0.8.9 on my VM and 0.6.5.9 on thinpower. Gemfile.lock specifies 0.9.0. We should consider using bundler on thinpower and my VM to use everywhere the same versions of the gems. I am not sure whether this would mean that we must first create clones for our patches gems!

import_daily produces cannot run swissindex-pharma

Added tracing the memory useage of the import, as my VM did not complete import_daily with less than 12 GB RAM. Diff see Attach:import_daily.txt. Memory useage was like this:

  • 2,40 GB before starting imports
  • 2,65 GB after update_swissmedic_feeds
  • 7,50 GB after update_textinfo_swissmedicinfo :fi
  • 7,27 GB after GC (explicit garbage collection)
  • 8,67 GB after update_textinfo_swissmedicinfo :pi
  • 8,50 GB after GC

My short term recommendation are:

  1. run textinfo_swissmedicinfo :fi and :pi as seperate jobs as they use an enormeous amout of memory.
  2. It might be worthwhile to spend a few hours to try to understand where all this memory is used and why it is never freed.
view · edit · sidebar · attach · print · history
Page last modified on October 15, 2013, at 08:21 AM