Running jobs/import_swissmedic with my local changes Attach:swissmedic-oddb-org-diff.txt I got the error
Plugin: ODDB::SwissmedicPlugin Error: ArgumentError Message: wrong number of arguments (1 for 2) Backtrace: /var/www/oddb.org/src/plugin/swissmedic.rb:158:in `cell' /var/www/oddb.org/src/plugin/swissmedic.rb:910:in `block in _sanity_check_deletions' /var/www/oddb.org/src/plugin/swissmedic.rb:910:in `delete_if' /var/www/oddb.org/src/plugin/swissmedic.rb:910:in `_sanity_check_deletions' /var/www/oddb.org/src/plugin/swissmedic.rb:904:in `sanity_check_deletions' /var/www/oddb.org/src/plugin/swissmedic.rb:93:in `update' /var/www/oddb.org/src/util/updater.rb:444:in `block in update_swissmedic' /var/www/oddb.org/src/util/updater.rb:549:in `call' /var/www/oddb.org/src/util/updater.rb:549:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:442:in `update_swissmedic' jobs/import_swissmedic:14:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_swissmedic:12:in `<module:Util>' jobs/import_swissmedic:11:in `<module:ODDB>' jobs/import_swissmedic:10:in `<main>'
Looking how I can fix it and whether it is in one of my skipped unit-tests. I find in test/test_plugin/swissmedic.rb methods like test__sanity_check_deletions test_sanity_check_deletions but there is so much mocking that they do not check the real code. Why? Found the culprit and it was easy to fix but I was unable to come up with a unittest in 10 minutes.
Looking at the log file I found 2015-07-06 23:41:04 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 86 Found 168 news, 17023 updates, 50 replacements and 149 package_deletions. There are many updates in the composition_text. Therefore checking the following entry http://ch.oddb.org/de/gcc?search_query=11745&search_type=st_registration&flavor=gcc&language=de&event=search which has Zusammensetzung : balsamum peruvianum 69 mg, ricini oleum virginale 345 mg, alcoholes adipis lanae acetylati, excipiens ad unguentum pro 1 g.
with the lines from the log
2015-07-06 23:41:19 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 883: res Rausch Perupomade, Salbe == Rausch Perupomade, Salbe? seqnr 01 args {:composition_text=>"balsamum peruvianum 69 mg, ricini oleum virginale 345 mg, alcoholes adipis lanae acetylati, excipiens ad unguentum pro 1 g.", :name_base=>"Rausch Perupomade", :name_descr=>"Salbe", :dose=>nil, :sequence_date=>#<Date: 1950-03-23 ((2433364j,0s,0n),+0s,2299161j)>, :export_flag=>nil}
2015-07-06 23:41:19 +0200: update_compositions: row[0] 11745 iksnr 11745 01 seq Rausch Perupomade, Salbe opts {:create_only=>false, :date=>#<Date: 2015-07-06 ((2457210j,0s,0n),+0s,2299161j)>} cell_content balsamum peruvianum 69 mg, ricini oleum virginale 345 mg, alcoholes adipis lanae acetylati, excipiens ad unguentum pro 1 g.
But it seems to have the same composition text and therefore no update should be necessary. Why? It gets updated regardless whether there was a change or not. Should we fix that? Okay. It looks like that comparing the expiration-date did not work correctly. E.g #rows_diff 00277 01 changed expiry_date: '#<Date: 2020-04-25 ((2458965j,0s,0n),+0s,2299161j)>' != '#<Date: 2015-04-25 ((2457138j,0s,0n),+0s,2299161j)>'
Fixed by adding next if left.is_a?(Date) and right.is_a?(Date) and left.start.eql?(right.start) in the method rows_diff of swissmedic-diff.rb.
Also removing the compatibility code for the the old XLS-file format. Now I think comparing the dates is okay.
Rerunning jobs/import_swissmedic with with packungen-latest from June and a restored Database from thinpower. Now I have 17026 updates, most of them because of the flag :indication_sequence. Why? Adding a binding.pry call to debug this problem in the import.
Packungen-2015.06.04.xls return the row R as an empty string "", whereas Packungen-2015.07.07.xlsx returns T Anwendungsgebiet Dosisstärke as (row.value) as nil. I probably did not catch this problems with my unit tests as I saved a smaller version of the Packungen using LibreOffice and MSOffice. NO! I just had not the correct expectations in my unit test! Import looks now a log better as it reports 2015-07-07 14:25:16 +0200: /var/www/oddb.org/src/plugin/swissmedic.rb: 86 Found 168 news, 439 updates, 50 replacements and 149 package_deletions. Waiting for import to finish.
Pushed commit for swissmedic-diff Fix comparing nil/empty strings and date cells
Job reported the following errors
Plugin: ODDB::Atc_lessPlugin Error: NoMethodError Message: undefined method `ITEM' for nil:NilClass Backtrace: /var/www/oddb.org/src/plugin/atc_less.rb:72:in `parse_refdata_xml' /var/www/oddb.org/src/plugin/atc_less.rb:98:in `update_atc_codes' /var/www/oddb.org/src/util/updater.rb:559:in `update_immediate' /var/www/oddb.org/src/util/updater.rb:454:in `update_atc_less' /var/www/oddb.org/src/util/updater.rb:457:in `update_swissmedic_followers' jobs/import_swissmedic:15:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_swissmedic:12:in `<module:Util>' jobs/import_swissmedic:11:in `<module:ODDB>' jobs/import_swissmedic:10:in `<main>'
and
Plugin: ODDB::SwissregPlugin Error: Net::HTTP::Persistent::Error Message: too many connection resets (due to Timeout::Error - Timeout::Error) after 312 requests on 15262680, last used 60.106963423 seconds ago Backtrace: (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/protocol.rb:132:in `readline' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:2551:in `read_new' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `catch' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/net/http.rb:1293:in `request' (druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:999:in `request' (druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize/http/agent.rb:259:in `fetch' (druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize.rb:1281:in `post_form' (druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize.rb:548:in `submit' (druby://localhost:10007) /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.7.3/lib/mechanize/form.rb:223:in `submit' (druby://localhost:10007) /var/www/oddb.org/ext/swissreg/src/swissreg.rb:85:in `search' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop' (druby://localhost:10007) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop' /var/www/oddb.org/src/plugin/swissreg.rb:58:in `update_registrations' /var/www/oddb.org/src/plugin/swissreg.rb:52:in `block in update_news' /var/www/oddb.org/src/plugin/swissreg.rb:49:in `each_key' /var/www/oddb.org/src/plugin/swissreg.rb:49:in `update_news' /var/www/oddb.org/src/util/updater.rb:559:in `update_immediate' /var/www/oddb.org/src/util/updater.rb:472:in `update_swissreg_news' /var/www/oddb.org/src/util/updater.rb:460:in `update_swissmedic_followers' jobs/import_swissmedic:15:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_swissmedic:12:in `<module:Util>' jobs/import_swissmedic:11:in `<module:ODDB>' jobs/import_swissmedic:10:in `<main>'
But the update of swissmedic seems okay, as I did get
ODDB::SwissmedicPlugin - Report 07.07.2015 Created Packages: 168 Updated Packages: 439 Deleted Packages: 149 (50 Replaced) Deactivated Sequences: 37 Deactivated Registrations: 24 Updated new Export-Registrations: 0 Updated existing Export-Registrations: 0 Updated new Export-Sequences: 0 Updated existing Export-Sequences: 0 Skipped Packages: 0 Total time to update: 79.00 [m] Total Sequences without ATC-Class: 16 Anzahl Sequenzen mit leerem Feld Zusammensetzung: 0 http://ch.oddb.apache.org/de/gcc/show/reg/00000/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/00828/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/10754/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/12498/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/17644/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/31919/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/42180/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/44978/seq/01 http://ch.oddb.apache.org/de/gcc/show/reg/52265/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/58458/seq/00 http://ch.oddb.apache.org/de/gcc/show/reg/58943/seq/01 http://ch.oddb.apache.org/de/gcc/show/reg/63051/seq/01 http://ch.oddb.apache.org/de/gcc/show/reg/63056/seq/01 http://ch.oddb.apache.org/de/gcc/show/reg/63242/seq/01 http://ch.oddb.apache.org/de/gcc/show/reg/65065/seq/01 http://ch.oddb.apache.org/de/gcc/show/reg/72736/seq/00
Pushed commit Adapat swissmedic plugin to new XLSX format
jobs/update_atc_less failed, because it got the wrong field from Packungen-latest.xlsx and because it failed to use the new RefdataPharma.xml file. Import is running again on oddb-ci2 and output looks fine till now.
Pushed commit Fixed atc_cless