view · edit · sidebar · attach · print · history

Index>

20141203-fix-bag-import-atc-code

Summary

  • Fix bag import. ATC-Codes should come from Packungen.xls of swissmedic
  • Add fields business_area and narcotics code to detail view of drug stores.

Commits

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!

---

Add fields business_area and narcotics code to detail view of drug stores.

Added these fields. The button correct does not work. But should we include it or not?

Pushed commit Added business_area/narcotics for pharmacies

Fix BAG XML import. ATC-Codes should come from Packungen.xls of swissmedic

jobs/import_bsv completed, but could not send mail via smtp. Why? Looking in log/oddb/debug/2014/12.log found 2014-12-02 20:25:39 CET /var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "admin@ywesee.com"

The ATC-code must come from the swissmedic Packungen.xls (Column F). But care has to be taken as they sometimes refer to a ATC-Code with less than 7 digits (e.g. IKSNR 274, Cardio-Pulmo-Rénal Sérocytol, suppositoire has J06AA). Also we might check whether it corresponds to content of the FI, e.g ATC-Code: J06AA)

atc_codes in oddb.org have german descriptions I don't know where they come from.

Updating in the plugin/swissmedic.rb does not work as expected unique_atc_class('R07AX') in bin/admin returns nil. atc_class('J06AA') works fine. Rebuilding substance_index_atc to see whether this changes.

Checking the fachinfo could be done with a snippet like this 

registration('62630').fachinfo.description(:de).effects.to_s.match(/ATC-Code: (\w+)/)[1]

With a (local) commit that should fix the problem I restarted the import_daily, which gets stuck and I see as last message on the standard output message type 0x5a arrived from server while idle. Googling around using ruby "message type 0x5a arrived from server while idle" I looks as this comes from DRB/Postgresql problem. See

Stopped all oddb-processes, restarted postgresql and starting import_daily another time.

Update fachinfo completed sucessfully. Many patinfos were also updated, but then I got the error 

  -> ptr false languages [:de] reg.iksnr 58544
2014-12-03 14:29:44 +0100: update_patinfo.pointer2 58544 :!patinfo,28750760.
2014-12-03 14:29:44 +0100: update_patinfo Hyperiplant® Rx iksnr 58544 update
2014-12-03 14:29:46 +0100: parse_and_update: calls parse_patinfo dist /var/www/oddb.org/data/html/patinfo/de/Hemeran_ Cr_me, Emulgel, Gel_swissmedicinfo.html iksnrs_from_xml ["29245", "37532", "49866"] Hemeran_ Cr_me, Emulgel, Gel_swissmedicinfo.html, name Hemeran® Crème, Emulgel, Gel de title Hemeran® Crème, Emulgel, Gel
2014-12-03 14:29:46 +0100: update_patinfo Hemeran® Crème, Emulgel, Gel iksnrs_from_xml ["29245", "37532", "49866"] empty false
2014-12-03 14:29:46 +0100: update_patinfo.pointer1 29245 
Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: Fach- und Patienteninfo Updates (swissmedicinfo.ch) - 12/2014 size 2409 with 0 attachments. Plugin: ODDB::TextInfoPlugin
Error: NotImplementedError
Message: method `method_missing' called on unexpected T_NODE object (0x00000052275b80 flags=0x21c klass=0x0)
Backtrace:
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:342:in `block in odba_replace_stubs'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:340:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:340:in `odba_replace_stubs'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:52:in `odba_receiver'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/plugin/text_info.rb:150:in `store_patinfo'
/var/www/oddb.org/src/plugin/text_info.rb:230:in `block in update_patinfo'
/var/www/oddb.org/src/plugin/text_info.rb:220:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:220:in `update_patinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1126:in `parse_and_update'
/var/www/oddb.org/src/plugin/text_info.rb:1136:in `block (3 levels) in import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1135:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1135:in `block (2 levels) in import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1134:in `each_pair'
/var/www/oddb.org/src/plugin/text_info.rb:1134:in `block in import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1131:in `each_pair'
/var/www/oddb.org/src/plugin/text_info.rb:1131:in `import_info'
/var/www/oddb.org/src/plugin/text_info.rb:1339:in `block in import_swissmedicinfo_by_index'
/var/www/oddb.org/src/plugin/text_info.rb:1338:in `each_pair'
/var/www/oddb.org/src/plugin/text_info.rb:1338:in `import_swissmedicinfo_by_index'
/var/www/oddb.org/src/plugin/text_info.rb:1393:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:550:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:544:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:336:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:194:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'
/var/www/oddb.org/log/job.pid is deleted

Restarting import as I cannot find any reason, why the import should fail here. Did not change text_info.rb in the last two months.

Now I got the same error as on thinpower 

Error: RuntimeError
Message: Unknown child node [mc:AlternateContent] for element [oleObjects]
Backtrace:
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/ooxml_object.rb:154:in `block in parse'
/usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node_set.rb:237:in `block in each'
/usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node_set.rb:236:in `upto'
/usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node_set.rb:236:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/ooxml_object.rb:143:in `parse'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/ooxml_object.rb:155:in `block in parse'
/usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node_set.rb:237:in `block in each'
/usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node_set.rb:236:in `upto'
/usr/local/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.3.1/lib/nokogiri/xml/node_set.rb:236:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/ooxml_object.rb:143:in `parse'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/worksheet.rb:731:in `parse'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/ooxml_object.rb:425:in `block in parse_file'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry.rb:500:in `get_input_stream'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/ooxml_object.rb:425:in `parse_file'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:90:in `block in load_related_files'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:75:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:75:in `load_related_files'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:172:in `load_relationships'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:91:in `block in load_related_files'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:75:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:75:in `load_related_files'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/relationships.rb:172:in `load_relationships'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/root.rb:68:in `block in parse_file'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:99:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/objects/root.rb:64:in `parse_file'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/parser.rb:18:in `parse'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyXL-3.2.7/lib/rubyXL/parser.rb:8:in `parse'
/usr/local/lib/ruby/gems/1.9.1/gems/swissmedic-diff-0.1.8/lib/compatibility.rb:10:in `open'
/var/www/oddb.org/src/plugin/swissmedic.rb:314:in `initialize_export_registrations'
/var/www/oddb.org/src/plugin/swissmedic.rb:68:in `update'
/var/www/oddb.org/src/util/updater.rb:424:in `block in update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:422:in `update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:197:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'
rm -f /var/www/oddb.org/data/xml/XMLPublications-2014.12.03.zip

Trying to reproduce the problem with the following snippet 

require 'swissmedic-diff'
latest_name = 'data/xls/Packungen-2014.12.03.xlsx'
puts "Opening #{latest_name}"
Spreadsheet.open(latest_name) do |workbook|
  workbook.worksheet(0).each(rows_to_skip(workbook)) do |row|
    iksnr = "%05i" % row[iksnr_idx].to_i
    puts iksnrt
  end
end

Works the same way and gives also the error ooxml_object.rb:154:in `block in parse': Unknown child node

Updating from rubyXL 3.2.7 to 3.3.0 did not fix the problem. But may the last commit 4 days ago as reported by https://github.com/weshatheleopard/rubyXL/issues/82 might help. Cloned the reposityr the code and executed there 

bundle install
bundle exec rake version:bump:patch
bundle exec rake build
sudo /usr/local/bin/gem install pkg/rubyXL-3.3.1.gem

But the error persisted. Opened https://github.com/weshatheleopard/rubyXL/issues/163 and trying to fix it locally. Opening the file and saving it under another name (Zugelassene_Packungen_problem.xlsx) using LibreOffice change the size from 2568181 to 2113687 bytes and the problem went away. Which means, I cannot produce a smaller file for the issue. Tried to detect the file type under bash with 

Zugelassene_Packungen_301114.xlsx:  Microsoft Excel 2007+
Zugelassene_Packungen_problem.xlsx: Microsoft OOXML
data/xls/Packungen-2014.10.16.xlsx: Microsoft Excel 2007+
data/xls/Packungen-2014.11.18.xlsx: Microsoft Excel 2007+

Which means, that saving files under libreoffice as XLSX they get silently other types.

Adapting get_latest to not saved the download if the file with the timestamp already exists. Must also convert Präparateliste.xlsx. This problem is solved, but I get another error 

Message: New column 18 ()
Backtrace:
/usr/local/lib/ruby/gems/1.9.1/gems/swissmedic-diff-0.1.8/lib/swissmedic-diff.rb:106:in `diff'
/var/www/oddb.org/src/plugin/swissmedic.rb:72:in `update'
/var/www/oddb.org/src/util/updater.rb:424:in `block in update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:422:in `update_swissmedic'
jobs/import_swissmedic:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_swissmedic:12:in `<module:Util>'
jobs/import_swissmedic:11:in `<module:ODDB>'
jobs/import_swissmedic:10:in `<main>'

Used on oddb-ci2 the following patch to circumvent this problem 

diff --git a/lib/swissmedic-diff.rb b/lib/swissmedic-diff.rb
index 86b5230..97b8861 100644
--- a/lib/swissmedic-diff.rb
+++ b/lib/swissmedic-diff.rb
@@ -103,7 +103,7 @@ class SwissmedicDiff
       tbook = Spreadsheet.open(target)
       sheet = tbook.worksheet(0)
       if new_column = cell(sheet.row(2), COLUMNS.size)
-        raise "New column #{COLUMNS.size} (#{new_column})"
+        $stdout.puts "New column #{COLUMNS.size} (#{new_column}) in #{target} is #{new_column.inspect}"
       end
       idx, prr, prp = nil
       multiples = {}

With this patch I restarted jobs/import_swissmedic and it began to treat the differences. After using almost 6 minutes of CPU time it suddenly blocked for over 12 minutes now. Last output was: 

2014-12-03 17:12:31 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb: 855: res Calperos D3, Lutschtabletten Mint == ? seqnr nil args {:composition_text=>"calcium 500 mg ut calcii carbonas, cholecalciferolum 400 U.I., aromatica, aspartamum, excipiens pro compresso.", :name_base=>"Calperos D3", :name_descr=>"Lutschtabletten Mint", :dose=>nil, :sequence_date=>#<DateTime: 2014-11-05T00:00:00+00:00 ((2456967j,0s,0n),+0s,2299161j)>, :export_flag=>nil, :atc_class=>"A12AX"}

Grepping whether atc_codes get updated resulted in 

2014-12-03 17:12:08 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb: 846: update_sequence atc_code_swissmedic D11AA atc_code_sequence D11AA equal true
2014-12-03 17:12:11 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb: 846: update_sequence atc_code_swissmedic L01BC02 atc_code_sequence L01BC02 equal true
2014-12-03 17:12:15 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb: 846: update_sequence atc_code_swissmedic J05AB01 atc_code_sequence J05AB01 equal true
2014-12-03 17:12:18 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb: 846: update_sequence atc_code_swissmedic B03BA51 atc_code_sequence B03BA53 equal false
2014-12-03 17:12:21 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb: 846: update_sequence atc_code_swissmedic B03BA51 atc_code_sequence B03BA53 equal false
2014-12-03 17:12:24 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb: 846: update_sequence atc_code_swissmedic J04AB02 atc_code_sequence J04AB02 equal true
view · edit · sidebar · attach · print · history
Page last modified on December 03, 2014, at 05:46 PM