view · edit · sidebar · attach · print · history

20130902-export-doctors

<< | Index | >>


Summary

  • error in Swissindex Pharma
  • error in export doctors.csv
  • more cleanups in unit tests

Commits

Index


error in Swissindex Pharma

Got the following error:

Plugin: ODDB::SwissindexPharmaPlugin
Error: NoMethodError
Message: undefined method `each' for nil:NilClass
Backtrace:
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:396:in `check_item'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
(druby://thinpower:37072) /var/www/oddb.org/src/plugin/swissindex.rb:103:in `block (2 levels) in update_package_trade_status'
(druby://thinpower:37072) /var/www/oddb.org/src/model/sequence.rb:175:in `each'
(druby://thinpower:37072) /var/www/oddb.org/src/model/sequence.rb:175:in `each_package'
(druby://thinpower:37072) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
(druby://thinpower:37072) /var/www/oddb.org/src/model/registration.rb:149:in `block in each_package'
(druby://thinpower:37072) /var/www/oddb.org/src/model/registration.rb:148:in `each_value'
(druby://thinpower:37072) /var/www/oddb.org/src/model/registration.rb:148:in `each_package'
(druby://thinpower:37072) /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
(druby://thinpower:37072) /var/www/oddb.org/src/util/oddbapp.rb:635:in `block in each_package'
(druby://thinpower:37072) /var/www/oddb.org/src/util/oddbapp.rb:634:in `each_value'
(druby://thinpower:37072) /var/www/oddb.org/src/util/oddbapp.rb:634:in `each_package'
(druby://thinpower:37072) /var/www/oddb.org/src/plugin/swissindex.rb:90:in `block in update_package_trade_status'
(druby://thinpower:37072) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `call'
(druby://thinpower:37072) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://thinpower:37072) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://thinpower:37072) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://thinpower:37072) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://thinpower:37072) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/invokemethod.rb:10:in `block_yield'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/invokemethod.rb:17:in `block in perform_with_block'
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:14:in `session'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/invokemethod.rb:14:in `perform_with_block'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1506:in `perform'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/swissindex.rb:88:in `update_package_trade_status'
/var/www/oddb.org/src/util/updater.rb:512:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:488:in `call'
/var/www/oddb.org/src/util/updater.rb:488:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:506:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:423:in `update_package_trade_status_by_swissindex'
jobs/update_swissindex_pharma:16:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/update_swissindex_pharma:13:in `<module:Util>'
jobs/update_swissindex_pharma:11:in `<module:ODDB>'
jobs/update_swissindex_pharma:10:in `<main>'

Running update_swissindex_pharma on my VM to see whether I can reproduce the behaviour here.

Got the following error while running

Plugin: ODDB::SwissindexPharmaPlugin
Error: NoMethodError
Message: undefined method `configure' for Savon:Module
Backtrace:
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:34:in `initialize'
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:14:in `new'
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:14:in `session'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/invokemethod.rb:14:in `perform_with_block'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1506:in `perform'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/swissindex.rb:88:in `update_package_trade_status'
/var/www/oddb.org/src/util/updater.rb:512:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:488:in `call'
/var/www/oddb.org/src/util/updater.rb:488:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:506:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:423:in `update_package_trade_status_by_swissindex'
/var/www/oddb.org/jobs/update_swissindex_pharma:16:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
/var/www/oddb.org/jobs/update_swissindex_pharma:13:in `<module:Util>'
/var/www/oddb.org/jobs/update_swissindex_pharma:11:in `<module:ODDB>'
/var/www/oddb.org/jobs/update_swissindex_pharma:10:in `<main>'

On thinpower I found

  • /usr/local/lib64/ruby/gems/1.9.1/gems/savon-0.9.7/lib/savon.rb.

On my VM I found

  • /home/vagrant/.rvm/gems/ruby-1.9.3-p0/gems/savon-0.8.6/lib/savon.rb
  • /usr/local/lib64/ruby/gems/1.9.1/gems/savon-2.1.0/lib/savon.rb

Replacing Savon 2.1.0 by 0.9.7 by changing the Gemfile and running "sudo bundle update". Restarted all oddb services. Then running import again. Now I get the following error

Plugin: ODDB::SwissindexPharmaPlugin
Error: DRb::DRbConnError
Message: druby://oddb.niklaus.org:60667 - #<Errno::ETIMEDOUT: Connection timed out - connect(2)>
Backtrace:
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:736:in `rescue in block in open'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:730:in `block in open'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:729:in `each'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:729:in `open'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1191:in `initialize'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1171:in `new'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1171:in `open'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1087:in `block in method_missing'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1105:in `with_friend'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1086:in `method_missing'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/invokemethod.rb:10:in `block_yield'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/invokemethod.rb:17:in `block in perform_with_block'
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:14:in `session'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/invokemethod.rb:14:in `perform_with_block'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1506:in `perform'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:50001) /usr/lib64/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/swissindex.rb:88:in `update_package_trade_status'
/var/www/oddb.org/src/util/updater.rb:512:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:488:in `call'
/var/www/oddb.org/src/util/updater.rb:488:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:506:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:423:in `update_package_trade_status_by_swissindex'
/var/www/oddb.org/jobs/update_swissindex_pharma:16:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
/var/www/oddb.org/jobs/update_swissindex_pharma:13:in `<module:Util>'
/var/www/oddb.org/jobs/update_swissindex_pharma:11:in `<module:ODDB>'
/var/www/oddb.org/jobs/update_swissindex_pharma:10:in `<main>'

Corrected host host/ip in /etc/hosts and rerunning import. The import did run without problems. See Attach:swissindex.txt

Will drop the PG-DB and import yesterday's import. After running the import I still don't get an error. Weird!

Installed via rvm (as user root into /usr/local/rvm) ruby-1.9.3.p0. To run it I set (logged in as root) the PATH to include /usr/local/rvm/bin and then sudo -u apache /usr/local/rvm/rubies/ruby-1.9.3-p0/bin/ruby /var/www/oddb.org/jobs/update_swissindex_pharma. But no I get errors because my RVM installation does not have the ruby patches applies (csv, columninfo, etc). Applying the patches from /usr/local/share/*.patch manually. Now the job begins to run. Completed again without any error! Don't know how to provoke it.

fix error with doctors.csv

Got the following error:

Error: ArgumentError
Message: invalid byte sequence in US-ASCII
Backtrace:
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/csv.rb:2071:in `=~'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/csv.rb:2071:in `init_separators'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/csv.rb:1603:in `initialize'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/csv.rb:1362:in `new'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/csv.rb:1362:in `open'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/csv_exporter.rb:98:in `dump'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:105:in `block (2 levels) in export_doc_csv'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:103:in `each'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:103:in `block in export_doc_csv'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `call'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:286:in `block in safe_export'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/tempfile.rb:316:in `open'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:285:in `safe_export'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:99:in `export_doc_csv'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1548:in `perform_without_block'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1508:in `perform'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:10005) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/csv_export.rb:25:in `export_doctors'
/var/www/oddb.org/src/util/exporter.rb:104:in `block in export_doc_csv'
/var/www/oddb.org/src/util/exporter.rb:435:in `call'
/var/www/oddb.org/src/util/exporter.rb:435:in `safe_export'
/var/www/oddb.org/src/util/exporter.rb:102:in `export_doc_csv'
/var/www/oddb.org/src/util/exporter.rb:58:in `run'
/var/www/oddb.org/jobs/export_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
/var/www/oddb.org/jobs/export_daily:12:in `<module:Util>'
/var/www/oddb.org/jobs/export_daily:11:in `<module:ODDB>'
/var/www/oddb.org/jobs/export_daily:10:in `<main>'

more cleanups in unit tests

Removed 'mock' in favor of 'flexmock'. See commit Removed mock in favor of flexmock

There were quite few problems because we used test-unit instead of test/unit. Also we have to exclude 5 test files as they create problems, most of them use 'util/session'. See commit Use test/unit instead of test-unit. Skip some problematic files

Now calling ruby test/suite.rb gives a result like

2699 tests, 3977 assertions, 169 failures, 336 errors, 0 pendings, 0 omissions, 17 notifications
83.8829% passed

32.87 tests/s, 48.43 assertions/s
Coverage report generated for Unit Tests to /opt/src/oddb.org/coverage. 22482 / 33886 LOC (66.35%) covered.

The skipped files show up in the log as

Skipping rwv2 dependent tests. Obsolete!!
Skipping bsv_xml csv_export
Skipping file bsv_xml.rb
Skipping file csv_export.rb
skipping ipn oddbapp session updater
Skipping file ipn.rb
Skipping file oddbapp.rb
Skipping file session.rb
Skipping file updater.rb

Reworked test/suite.rb to generate correct overall coverage info and a status and the end. Manuall grepping the log shows that we still have some errors and failures.

239 tests, 593 assertions, 17 failures, 21 errors, 0 pendings, 0 omissions, 0 notifications
796 tests, 816 assertions, 54 failures, 83 errors, 0 skips
271 tests, 299 assertions, 33 failures, 23 errors, 1 skips
3 tests, 3 assertions, 0 failures, 0 errors, 0 skips
6 tests, 6 assertions, 1 failures, 0 errors, 0 skips
43 tests, 43 assertions, 10 failures, 0 errors, 0 skips
446 tests, 658 assertions, 14 failures, 72 errors, 0 skips
661 tests, 1380 assertions, 22 failures, 1 errors, 0 skips
312 tests, 457 assertions, 17 failures, 6 errors, 0 skips

But it is important, that we run each test/suite.rb independent of each other as running them all in one run induces errors, which are not present when run individually! E.g I want to be sure that "grep ^Error: suite.log | grep galenic" does not return anything.

Corrections made with these commits:

view · edit · sidebar · attach · print · history
Page last modified on September 11, 2013, at 02:02 PM