could not connect to www.swissreg.ch: #<Net::HTTPInternalServerError:0x007f8a7d69bb58>
---
Import-BSV returns without doing anything after downloading the file. Running jobs/import_bsv generates the following log messages under log/oddb/debug/2014/02.log
2014-02-03 09:36:53 CET safe_export Mail Patinfo-Invoices starting 2014-02-03 09:37:52 CET getin update_bsv 2014-02-03 09:38:00 CET getin BsvXmlPlugin.update 2014-02-03 09:38:00 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip 2014-02-03 09:38:00 CET save_dir = /var/www/oddb.org/data/xml 2014-02-03 09:38:00 CET getin download_file 2014-02-03 09:38:02 CET save_file = /var/www/oddb.org/data/xml/XMLPublications-2014.02.03.zip 2014-02-03 09:38:02 CET latest_file = /var/www/oddb.org/data/xml/XMLPublications-latest.zip 2014-02-03 09:38:02 CET File.exists?(/var/www/oddb.org/data/xml/XMLPublications-latest.zip) = true 2014-02-03 09:38:02 CET FileUtils.compare_file(/var/www/oddb.org/data/xml/XMLPublications-2014.02.03.zip, /var/www/oddb.org/data/xml/XMLPublications-latest.zip) = false 2014-02-03 09:38:02 CET path = "/var/www/oddb.org/data/xml/XMLPublications-2014.02.03.zip" 2014-02-03 09:38:02 CET entry.name = Gestrichene_Packungen_Emballages_radies.xls 2014-02-03 09:38:02 CET entry.name = GL_Diff_SB.xml 2014-02-03 09:38:02 CET entry.name = ItCodes.xml
Looking at the size of the downloaded files using ls -lrt /var/www/oddb.org/data/xml/XMLPublications* on thinpower I find
-rw-r--r-- 1 apache apache 4640044 1. Jan 09:20 /var/www/oddb.org/data/xml/XMLPublications-2014.01.01.zip -rw-r--r-- 1 apache apache 4645637 15. Jan 09:30 /var/www/oddb.org/data/xml/XMLPublications-2014.01.15.zip -rw-r--r-- 1 apache apache 4744707 1. Feb 22:20 /var/www/oddb.org/data/xml/XMLPublications-2014.02.01.zip -rw-r--r-- 1 apache apache 4744707 1. Feb 22:20 /var/www/oddb.org/data/xml/XMLPublications-latest.zip
Looking whether the change of size size on Februry 1 resulted in a ran grep return_value_update_bsv /var/www/oddb.org/log/oddb/debug/2014/02.log
and got the following answer
2014-02-01 09:18:50 CET return_value_update_bsv=nil 2014-02-02 09:19:29 CET return_value_update_bsv=nil 2014-02-03 09:18:09 CET return_value_update_bsv=nil
Therefore I must correct the comparison. This seems to work. But the probems lies that calling update_it_codes exits the program without any error messages. Why?
Added a rescue clause in src/plugin/bsv_xml.rb (_update) to get the callback
Plugin: ODDB::BsvXmlPlugin Error: NoMethodError Message: undefined method `pos' for #<Zip::ZipInputStream:0x007fa6227d2a10> Backtrace: /usr/lib64/ruby/1.9.1/rexml/source.rb:236:in `current_line' /usr/lib64/ruby/1.9.1/rexml/parseexception.rb:44:in `line' /usr/lib64/ruby/1.9.1/rexml/parseexception.rb:28:in `to_s' /var/www/oddb.org/src/plugin/bsv_xml.rb:670:in `inspect' /var/www/oddb.org/src/plugin/bsv_xml.rb:670:in `rescue in _update' /var/www/oddb.org/src/plugin/bsv_xml.rb:654:in `_update' /var/www/oddb.org/src/plugin/bsv_xml.rb:649:in `update' /var/www/oddb.org/src/util/updater.rb:247:in `block in update_bsv' /var/www/oddb.org/src/util/updater.rb:495:in `call' /var/www/oddb.org/src/util/updater.rb:495:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:245:in `update_bsv' jobs/import_bsv:14:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_bsv:12:in `<module:Util>' jobs/import_bsv:11:in `<module:ODDB>' jobs/import_bsv:10:in `<main>'
Trying to use pry to examine the problem.
Tried using rubyzip 0.9.4 like rubygem on thinpower. This did not change at all.
Single-stepping in bsv_xml with pry produced:
From: /home/niklaus/.rvm/rubies/ruby-1.9.3-rc1/lib/ruby/1.9.1/rexml/parsers/streamparser.rb @ line 16 REXML::Parsers::StreamParser#parse:
11: end
12:
13: def parse
14: # entity string
15: while true
=> 16: event = @parser.pull
17: case event[0]
18: when :end_document
19: return
20: when :start_element
21: attrs = event[2].each do |n, v|
[6] pry(#<REXML::Parsers::StreamParser>)> next
From: /var/www/oddb.org/src/plugin/bsv_xml.rb @ line 1020 ODDB::BsvXmlPlugin#update_it_codes:
1016: def update_it_codes io
1017: listener = ItCodesListener.new @app
1018: REXML::Document.parse_stream io, listener
1019: rescue REXML::ParseException => e
=> 1020: require 'pry'; binding.pry
1021: end
Why has parser.pull problems scanning this document? Single Stepping and looking at the error when rescue I got a good explanation
15] pry(#<REXML::Parsers::BaseParser>)> caller => ["/var/www/oddb.org/vendor/bundle/ruby/1.9.1/gems/pry-0.9.12.4/lib/pry/pry_instance.rb:328:in `eval'", <...> "/home/niklaus/.rvm/rubies/ruby-1.9.3-rc1/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:439:in `rescue in pull'", "/home/niklaus/.rvm/rubies/ruby-1.9.3-rc1/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:330:in `pull'", "/home/niklaus/.rvm/rubies/ruby-1.9.3-rc1/lib/ruby/1.9.1/rexml/parsers/streamparser.rb:16:in `parse'", [16] pry(#<REXML::Parsers::BaseParser>)> error => #<Encoding::CompatibilityError: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)>
Seems as we had similar problems when migration to 1.9.3. See http://dev.ywesee.com/Choddb/Ruby193p0. Are they back? If so, why???
After reapplying the patches baseparser.rb.patch and source.rb.patch the import_bsv worked fine, but has some curious messages like
Could not create: :!registration,51164!sequence,01!016!sl_entry., reason: :!registration,51164!sequence,01!016!sl_entry. -> ODDB::Sequence::package(016) returned nil Could not create: :!registration,51164!sequence,01!024!sl_entry., reason: :!registration,51164!sequence,01!024!sl_entry. -> ODDB::Sequence::package(024) returned nil ODBA::Stub was unable to replace ODDB::SlEntry#705477 from ODDB::Package:#226260 ODBA::Stub was unable to replace ODDB::SlEntry#705476 from ODDB::Package:#226259
It looks that not everything is consistent, e.g bin/admin reports
atc_class('J01AA08').packages.collect{|x| x.iksnr}
-> ["37230", "45465", "45465", "53783", "52940", "52940", "58135"]
registration('51164').atc_classes[0].code
-> J01AA08
Whereas Preparation.xml has the following definition
<Preparation ProductCommercial="192">
<NameDe>Aknoral</NameDe>
<NameFr>Aknoral</NameFr>
<NameIt>Aknoral</NameIt>
<DescriptionDe>Caps 50 mg </DescriptionDe>
<DescriptionFr>caps 50 mg </DescriptionFr>
<DescriptionIt>caps 50 mg </DescriptionIt>
<AtcCode>J01AA08</AtcCode>
<SwissmedicNo5>51164</SwissmedicNo5>
<FlagItLimitation>Y</FlagItLimitation>
<OrgGenCode>G</OrgGenCode>
<...>
List of all lines containing sl_entry., is in Attach:sl_entries.txt. The bsv-import produced still an error Error Export: oddb2tdat with migel namely
Error: Errno::ENOENT Message: No such file or directory - /var/www/migel/data/csv/migel_product_de.csv Backtrace: /var/www/oddb.org/vendor/bundle/ruby/1.9.1/gems/oddb2tdat-1.1.2/lib/oddb2tdat.rb:175:in `initialize' /var/www/oddb.org/vendor/bundle/ruby/1.9.1/gems/oddb2tdat-1.1.2/lib/oddb2tdat.rb:175:in `open' /var/www/oddb.org/vendor/bundle/ruby/1.9.1/gems/oddb2tdat-1.1.2/lib/oddb2tdat.rb:175:in `parse_migel' /var/www/oddb.org/vendor/bundle/ruby/1.9.1/gems/oddb2tdat-1.1.2/lib/oddb2tdat.rb:93:in `run' /var/www/oddb.org/src/plugin/csv_export.rb:202:in `export_oddb_dat_with_migel' /var/www/oddb.org/src/util/exporter.rb:236:in `block in export_oddb2tdat_with_migel' /var/www/oddb.org/src/util/exporter.rb:425:in `call' /var/www/oddb.org/src/util/exporter.rb:425:in `safe_export' /var/www/oddb.org/src/util/exporter.rb:234:in `export_oddb2tdat_with_migel' /var/www/oddb.org/src/util/updater.rb:112:in `export_oddb2tdat_with_migel' /var/www/oddb.org/src/util/updater.rb:267:in `update_bsv_followers' jobs/import_bsv:15:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_bsv:12:in `<module:Util>' jobs/import_bsv:11:in `<module:ODDB>' jobs/import_bsv:10:in `<main>'
The daily export gave the following error message
Error: ArgumentError Message: comparison of FalseClass with String failed Backtrace: (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:241:in `sort_by' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:241:in `block in export_price_history_csv' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `call' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `block in safe_export' (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/tempfile.rb:316:in `open' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:285:in `safe_export' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:219:in `export_price_history_csv' (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block' (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform' (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop' (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop' (druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop' /var/www/oddb.org/src/plugin/csv_export.rb:171:in `export_price_history' /var/www/oddb.org/src/util/exporter.rb:361:in `block in export_price_history_csv' /var/www/oddb.org/src/util/exporter.rb:425:in `call' /var/www/oddb.org/src/util/exporter.rb:425:in `safe_export' /var/www/oddb.org/src/util/exporter.rb:359:in `export_price_history_csv' /var/www/oddb.org/src/util/exporter.rb:75:in `run' jobs/export_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/export_daily:12:in `<module:Util>' jobs/export_daily:11:in `<module:ODDB>' jobs/export_daily:10:in `<main>'
export index_therapeuticus gives
Error: ArgumentError Message: comparison of NilClass with String failed Backtrace: /var/www/oddb.org/src/plugin/csv_export.rb:161:in `sort_by' /var/www/oddb.org/src/plugin/csv_export.rb:161:in `export_index_therapeuticus' /var/www/oddb.org/src/util/exporter.rb:169:in `block in export_index_therapeuticus_csv' /var/www/oddb.org/src/util/exporter.rb:425:in `call' /var/www/oddb.org/src/util/exporter.rb:425:in `safe_export' /var/www/oddb.org/src/util/exporter.rb:167:in `export_index_therapeuticus_csv' /var/www/oddb.org/src/util/exporter.rb:74:in `run' jobs/export_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/export_daily:12:in `<module:Util>' jobs/export_daily:11:in `<module:ODDB>' jobs/export_daily:10:in `<main>'
oddb2.csv gave
Error: NoMethodError Message: undefined method `first' for false:FalseClass Backtrace: /var/www/oddb.org/src/view/drugs/csv_result.rb:143:in `galenic_form' /var/www/oddb.org/src/view/drugs/csv_result.rb:150:in `galenic_form_de' /var/www/oddb.org/src/view/drugs/csv_result.rb:450:in `block (3 levels) in to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:448:in `collect' /var/www/oddb.org/src/view/drugs/csv_result.rb:448:in `block (2 levels) in to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:433:in `each' /var/www/oddb.org/src/view/drugs/csv_result.rb:433:in `block in to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:426:in `each' /var/www/oddb.org/src/view/drugs/csv_result.rb:426:in `to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:470:in `block in to_csv_file' /var/www/oddb.org/src/view/drugs/csv_result.rb:469:in `open' /var/www/oddb.org/src/view/drugs/csv_result.rb:469:in `to_csv_file' /var/www/oddb.org/src/plugin/csv_export.rb:63:in `_export_drugs' /var/www/oddb.org/src/plugin/csv_export.rb:42:in `export_drugs_extended' /var/www/oddb.org/src/util/exporter.rb:106:in `block in export_csv' /var/www/oddb.org/src/util/exporter.rb:425:in `call' /var/www/oddb.org/src/util/exporter.rb:425:in `safe_export' /var/www/oddb.org/src/util/exporter.rb:105:in `export_csv' /var/www/oddb.org/src/util/exporter.rb:72:in `run' jobs/export_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/export_daily:12:in `<module:Util>' jobs/export_daily:11:in `<module:ODDB>' jobs/export_daily:10:in `<main>'
and oddb.csv gave
Error: NoMethodError Message: undefined method `first' for false:FalseClass Backtrace: /var/www/oddb.org/src/view/drugs/csv_result.rb:143:in `galenic_form' /var/www/oddb.org/src/view/drugs/csv_result.rb:450:in `block (3 levels) in to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:448:in `collect' /var/www/oddb.org/src/view/drugs/csv_result.rb:448:in `block (2 levels) in to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:433:in `each' /var/www/oddb.org/src/view/drugs/csv_result.rb:433:in `block in to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:426:in `each' /var/www/oddb.org/src/view/drugs/csv_result.rb:426:in `to_csv' /var/www/oddb.org/src/view/drugs/csv_result.rb:470:in `block in to_csv_file' /var/www/oddb.org/src/view/drugs/csv_result.rb:469:in `open' /var/www/oddb.org/src/view/drugs/csv_result.rb:469:in `to_csv_file' /var/www/oddb.org/src/plugin/csv_export.rb:63:in `_export_drugs' /var/www/oddb.org/src/plugin/csv_export.rb:30:in `export_drugs' /var/www/oddb.org/src/util/exporter.rb:103:in `block in export_csv' /var/www/oddb.org/src/util/exporter.rb:425:in `call' /var/www/oddb.org/src/util/exporter.rb:425:in `safe_export' /var/www/oddb.org/src/util/exporter.rb:102:in `export_csv' /var/www/oddb.org/src/util/exporter.rb:72:in `run' jobs/export_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/export_daily:12:in `<module:Util>' jobs/export_daily:11:in `<module:ODDB>' jobs/export_daily:10:in `<main>'
Looking first at jobs/export_oddb2_csv to figure out why a galenic_form is no longer an array. Export completed without error on oddb-ci2.
Running export_daily completed without error on oddb-ci2.
Imported DB-dump of thinpower and restarted a complete jobs/export_daily at 21.10 hours. Will look at the results tomorrow.
Results are: Now I have problem exporting for oddb.csv, oddb2.csv, index_therapeuticus.csv and price_history.csv.