view · edit · sidebar · attach · print · history

Index>

20150707-swissmedic-oddb-org

Summary

  • Adapt oddb.org import swissmedic to changed format of Packungen-xlsx

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Display new fields (LABEL, MORE_INFO, CORRESP) for compositions in oddb.org.
  • Use refdatabase for oddb.org, too.
  • Fix running rspec for new packungen.xlsx in oddb2xml

Adapt oddb.org import swissmedic to changed format of Packungen-xlsx

Running jobs/import_swissmedic with my local changes Attach:swissmedic-oddb-org-diff.txt I got the error

Plugin: ODDB::SwissmedicPlugin
Error: ArgumentError
Message: wrong number of arguments (1 for 2)
Backtrace:
/var/www/oddb.org/src/plugin/swissmedic.rb:158:in `cell'
/var/www/oddb.org/src/plugin/swissmedic.rb:910:in `block in _sanity_check_deletions'
/var/www/oddb.org/src/plugin/swissmedic.rb:910:in `delete_if'
/var/www/oddb.org/src/plugin/swissmedic.rb:910:in `_sanity_check_deletions'
/var/www/oddb.org/src/plugin/swissmedic.rb:904:in `sanity_check_deletions'
/var/www/oddb.org/src/plugin/swissmedic.rb:93:in `update'
/var/www/oddb.org/src/util/updater.rb:444:in `block in update_swissmedic'
/var/www/oddb.org/src/util/updater.rb:549:in `call'
/var/www/oddb.org/src/util/updater.rb:549:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:442:in `update_swissmedic'
jobs/import_swissmedic:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_swissmedic:12:in `<module:Util>'
jobs/import_swissmedic:11:in `<module:ODDB>'
jobs/import_swissmedic:10:in `<main>'

Looking how I can fix it and whether it is in one of my skipped unit-tests. I find in test/test_plugin/swissmedic.rb methods like test__sanity_check_deletions test_sanity_check_deletions but there is so much mocking that they do not check the real code. Why? Found the culprit and it was easy to fix but I was unable to come up with a unittest in 10 minutes.

Looking at the log file I found 2015-07-06 23:41:04 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 86 Found 168 news, 17023 updates, 50 replacements and 149 package_deletions. There are many updates in the composition_text. Therefore checking the following entry http://ch.oddb.org/de/gcc?search_query=11745&search_type=st_registration&flavor=gcc&language=de&event=search which has Zusammensetzung : balsamum peruvianum 69 mg, ricini oleum virginale 345 mg, alcoholes adipis lanae acetylati, excipiens ad unguentum pro 1 g.

 with the lines from the log
2015-07-06 23:41:19 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 883: res Rausch Perupomade, Salbe == Rausch Perupomade, Salbe? seqnr 01 args {:composition_text=>"balsamum peruvianum 69 mg, ricini oleum virginale 345 mg, alcoholes adipis lanae acetylati, excipiens ad unguentum pro 1 g.", :name_base=>"Rausch Perupomade", :name_descr=>"Salbe", :dose=>nil, :sequence_date=>#<Date: 1950-03-23 ((2433364j,0s,0n),+0s,2299161j)>, :export_flag=>nil}
2015-07-06 23:41:19 +0200: update_compositions: row[0] 11745 iksnr 11745 01 seq Rausch Perupomade, Salbe opts {:create_only=>false, :date=>#<Date: 2015-07-06 ((2457210j,0s,0n),+0s,2299161j)>} cell_content balsamum peruvianum 69 mg, ricini oleum virginale 345 mg, alcoholes adipis lanae acetylati, excipiens ad unguentum pro 1 g.

But it seems to have the same composition text and therefore no update should be necessary. Why? It gets updated regardless whether there was a change or not. Should we fix that? Okay. It looks like that comparing the expiration-date did not work correctly. E.g #rows_diff 00277 01 changed expiry_date: '#<Date: 2020-04-25 ((2458965j,0s,0n),+0s,2299161j)>' != '#<Date: 2015-04-25 ((2457138j,0s,0n),+0s,2299161j)>'

Fixed by adding next if left.is_a?(Date) and right.is_a?(Date) and left.start.eql?(right.start) in the method rows_diff of swissmedic-diff.rb.

Also removing the compatibility code for the the old XLS-file format. Now I think comparing the dates is okay.

Rerunning jobs/import_swissmedic with with packungen-latest from June and a restored Database from thinpower. Now I have 17026 updates, most of them because of the flag :indication_sequence. Why? Adding a binding.pry call to debug this problem in the import.

Packungen-2015.06.04.xls return the row R as an empty string "", whereas Packungen-2015.07.07.xlsx returns T Anwendungsgebiet Dosisstärke as (row.value) as nil. I probably did not catch this problems with my unit tests as I saved a smaller version of the Packungen using LibreOffice and MSOffice. NO! I just had not the correct expectations in my unit test! Import looks now a log better as it reports 2015-07-07 14:25:16 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 86 Found 168 news, 439 updates, 50 replacements and 149 package_deletions. Waiting for import to finish.

Pushed commit for swissmedic-diff Fix comparing nil/empty strings and date cells

Job reported the following errors

Plugin: ODDB::Atc_lessPlugin
Error: NoMethodError
Message: undefined method `ITEM' for nil:NilClass
Backtrace:
/var/www/oddb.org/src/plugin/atc_less.rb:72:in `parse_refdata_xml'
/var/www/oddb.org/src/plugin/atc_less.rb:98:in `update_atc_codes'
/var/www/oddb.org/src/util/updater.rb:559:in `update_immediate'
/var/www/oddb.org/src/util/updater.rb:454:in `update_atc_less'
/var/www/oddb.org/src/util/updater.rb:457:in `update_swissmedic_followers'
jobs/import_swissmedic:15:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_swissmedic:12:in `<module:Util>'
jobs/import_swissmedic:11:in `<module:ODDB>'
jobs/import_swissmedic:10:in `<main>'

and

Plugin: ODDB::SwissregPlugin
Error: Net::HTTP::Persistent::Error
Message: too many connection resets (due to Timeout::Error - Timeout::Error) after 312 requests on 15262680, last used 60.106963423 seconds ago
Backtrace:
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1293:in `request'
(druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:999:in `request'
(druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:259:in `fetch'
(druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize.rb:1281:in `post_form'
(druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize.rb:548:in `submit'
(druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize/form.rb:223:in `submit'
(druby://localhost:10007) /var/www/oddb.org/ext/swissreg/src/swissreg.rb:85:in `search'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/swissreg.rb:58:in `update_registrations'
/var/www/oddb.org/src/plugin/swissreg.rb:52:in `block in update_news'
/var/www/oddb.org/src/plugin/swissreg.rb:49:in `each_key'
/var/www/oddb.org/src/plugin/swissreg.rb:49:in `update_news'
/var/www/oddb.org/src/util/updater.rb:559:in `update_immediate'
/var/www/oddb.org/src/util/updater.rb:472:in `update_swissreg_news'
/var/www/oddb.org/src/util/updater.rb:460:in `update_swissmedic_followers'
jobs/import_swissmedic:15:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_swissmedic:12:in `<module:Util>'
jobs/import_swissmedic:11:in `<module:ODDB>'
jobs/import_swissmedic:10:in `<main>'

But the update of swissmedic seems okay, as I did get

ODDB::SwissmedicPlugin - Report 07.07.2015
Created Packages: 168
Updated Packages: 439
Deleted Packages: 149 (50 Replaced)
Deactivated Sequences: 37
Deactivated Registrations: 24
Updated new Export-Registrations: 0
Updated existing Export-Registrations: 0
Updated new Export-Sequences: 0
Updated existing Export-Sequences: 0
Skipped Packages: 0
Total time to update: 79.00 [m]
Total Sequences without ATC-Class: 16
Anzahl Sequenzen mit leerem Feld Zusammensetzung: 0
http://ch.oddb.apache.org/de/gcc/show/reg/00000/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/00828/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/10754/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/12498/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/17644/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/31919/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/42180/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/44978/seq/01
http://ch.oddb.apache.org/de/gcc/show/reg/52265/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/58458/seq/00
http://ch.oddb.apache.org/de/gcc/show/reg/58943/seq/01
http://ch.oddb.apache.org/de/gcc/show/reg/63051/seq/01
http://ch.oddb.apache.org/de/gcc/show/reg/63056/seq/01
http://ch.oddb.apache.org/de/gcc/show/reg/63242/seq/01
http://ch.oddb.apache.org/de/gcc/show/reg/65065/seq/01
http://ch.oddb.apache.org/de/gcc/show/reg/72736/seq/00

Pushed commit Adapat swissmedic plugin to new XLSX format

jobs/update_atc_less failed, because it got the wrong field from Packungen-latest.xlsx and because it failed to use the new RefdataPharma.xml file. Import is running again on oddb-ci2 and output looks fine till now.

Pushed commit Fixed atc_cless

view · edit · sidebar · attach · print · history
Page last modified on July 07, 2015, at 06:49 PM