20110802-debug-slupdate-oddb_org-testcases-bbmb_ch

<< Masa.20110803-test-statistics-bbmb_ch-debug-price-comparison-patinfo-link-oddb_org | 2011 | Masa.20110729-setup-ddd-bbmb_ch >>

1. Change ATC DDD Version from 2004 to 2009
2. Debug SL-Update (XML) error
3. Testcases bbmb.ch
4. Testcases with the updates of new ids process
5. Check if run_statistics works

Goal/Estimate/Evaluation

• Debug SL-Update (XML) / 100% / 100%
• Testcases bbmb.ch / 80% / 100%

Milestones

Summary

• http://masay.ch/2011/08/02/debug-sl-update-process-of-oddb-org-and-fix-testcasese-of-bbmb-ch/

Commits

• Fixed an update_bsv bug, it seems that sometimes EOFError happens before the creating of a temp file for the downloaded xml file (oddb.org)
• Fixed swissindex plugin to catch a timeout error (oddb.org)

Change ATC DDD Version

• src/custom/lookandfeelbase.rb

    DICTIONARIES = {
      "de"  =>  {
        :ddd_version            =>  'ATC-DDD Version 2009. Source: ',

Debug SL-Update (XML) error

Email

• ch.ODDB.org Report - Error: SL-Update (XML) - 08/2011

Plugin: ODDB::BsvXmlPlugin
Error: NoMethodError
Message: undefined method `close' for nil:NilClass
Backtrace:
/var/www/oddb.org/src/plugin/bsv_xml.rb:688:in `download_file'
/var/www/oddb.org/src/plugin/bsv_xml.rb:627:in `update'
/var/www/oddb.org/src/util/updater.rb:251:in `update_bsv'
/var/www/oddb.org/src/util/updater.rb:473:in `call'
/var/www/oddb.org/src/util/updater.rb:473:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:249:in `update_bsv'
/var/www/oddb.org/src/util/updater.rb:212:in `run'
/var/www/oddb.org/jobs/import_daily:13
/var/www/oddb.org/src/util/job.rb:17:in `call'
/var/www/oddb.org/src/util/job.rb:17:in `run'
/var/www/oddb.org/jobs/import_daily:12

Confirm the error locally

Run

• bin/oddbd
• ext/swissindex/bin/swissindex_pharmad

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_bsv

Result

• Email ch.ODDB.org Report - Error: SL-Update (XML) - 08/2011

Plugin: ODDB::BsvXmlPlugin
Error: NoMethodError
Message: undefined method `close' for nil:NilClass
Backtrace:
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:688:in `download_file'
/home/masa/ywesee/oddb.org/src/plugin/bsv_xml.rb:627:in `update'
/home/masa/ywesee/oddb.org/src/util/updater.rb:251:in `update_bsv'
/home/masa/ywesee/oddb.org/src/util/updater.rb:473:in `call'
/home/masa/ywesee/oddb.org/src/util/updater.rb:473:in `wrap_update'
/home/masa/ywesee/oddb.org/src/util/updater.rb:249:in `update_bsv'
(eval):1:in `_admin'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `call'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `failsafe'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1433:in `_admin'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1431:in `initialize'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1431:in `new'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1431:in `_admin'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1515:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1589:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1430:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1347:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `start_service'
bin/oddbd:38

Note

• It is the same error

Experiment

• src/plugin/bsv_xml.rb#download_file

    ensure
      if temp
        temp.close
        temp.unlink
      end
    end

Result

• Looks working

Commit

• Fixed an update_bsv bug, it seems that sometimes EOFError happens before the creating of a temp file for the downloaded xml file (oddb.org)
• Fixed swissindex plugin to catch a timeout error (oddb.org)

Run online server

screen -S import_bsv
sudo -u apache ./jobs/import_bsv

Result

• Email https://mail.google.com/mail/?shva=1#inbox/131897c8ca05e91e success

Testcases bbmb.ch

  1) Failure:
test_create_hospital(BBMB::TestBbmbAppAdmin) [./test/test_bbmbapp.rb:392]:
<BBMB::HospitalUser> expected but was
<NilClass>.

  2) Failure:
test_generate_password(BBMB::TestBbmbAppAdmin) [./test/test_bbmbapp.rb:434]:
<true> expected but was
<false>.

  3) Failure:
test_save_hospital(BBMB::TestBbmbAppAdmin) [./test/test_bbmbapp.rb:400]:
<true> expected but was
<false>.

  4) Error:
test_dont_send_empty_order(BBMB::TestBbmbAppHospital):
NoMethodError: undefined method `value' for nil:NilClass
    /home/masa/work/bbmb.ch/src/transactions/stock.rb:16:in `execute'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:11:in `execute'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:221:in `execute'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:192:in `runAt'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:220:in `execute'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:26:in `executeTransaction'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:23:in `synchronize'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:23:in `executeTransaction'
    /home/masa/work/bbmb.ch/src/util/bbmbapp.rb:655:in `update_stock'
    ./test/test_bbmbapp.rb:494:in `setup'

  5) Error:
test_keep_price(BBMB::TestBbmbAppHospital):
NoMethodError: undefined method `value' for nil:NilClass
    /home/masa/work/bbmb.ch/src/transactions/stock.rb:16:in `execute'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:11:in `execute'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:221:in `execute'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:192:in `runAt'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:220:in `execute'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:26:in `executeTransaction'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:23:in `synchronize'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:23:in `executeTransaction'
    /home/masa/work/bbmb.ch/src/util/bbmbapp.rb:655:in `update_stock'
    ./test/test_bbmbapp.rb:494:in `setup'

  6) Error:
test_send_order(BBMB::TestBbmbAppHospital):
NoMethodError: undefined method `value' for nil:NilClass
    /home/masa/work/bbmb.ch/src/transactions/stock.rb:16:in `execute'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:11:in `execute'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:221:in `execute'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:192:in `runAt'
    /usr/lib64/ruby/site_ruby/1.8/mnemonic/Mnemonic.rb:220:in `execute'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:26:in `executeTransaction'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:23:in `synchronize'
    /home/masa/work/bbmb.ch/src/transactions/transactions.rb:23:in `executeTransaction'
    /home/masa/work/bbmb.ch/src/util/bbmbapp.rb:655:in `update_stock'
    ./test/test_bbmbapp.rb:494:in `setup'

  7) Failure:
test_resolve(BBMB::TestStock) [./test/test_stock.rb:95]:
<"10002"> expected but was
<"10001">.

187 tests, 1046 assertions, 4 failures, 3 errors

Patch

• Attach:bbmb_patch.20110802.txt

Memo (how to create a patch, apply a patch)

 bbmb.ch $ git diff > bbmb_patch

 bbmb.ch $ patch -p1 < bbmb_patch

Run

$ git clone git://scm.ywesee.com/bbmb.ch
$ cd bbmb.ch
$ patch -p1 < bbmb_patch
patching file test/test_bbmbapp.rb
patching file test/test_behavior.rb
patching file test/test_order.rb
patching file test/test_reseller.rb
patching file test/test_stock.rb
patching file test/test_user.rb
patching file test/test_validator.rb
$ ruby -W0 test/suite.rb
187 tests, 1546 assertions, 9 failures, 22 errors
$ ruby -W0 test/suite.rb
187 tests, 1774 assertions, 0 failures, 0 errors
$ ruby -W0 test/suite.rb
187 tests, 1784 assertions, 0 failures, 0 errors

Note

• The first test run returns many failures and errors, but they are gone in the second run
• And the number of assertions increases every run (10 assertions are added)
• The number of assertions in test_bbmbapp.rb increases
• It seems that some resellers and orders those are related to the number of assersions are created during the testcases

Question

• Where is the data saved while the testcases run?
• There is no log directory created.

Answer

masa@masa ~/work/test/bbmb.ch $ grep -r STORAGE_PATH test/
test/stub_bbmb.rb:      STORAGE_PATH = "/tmp/bbmbtest"
masa@masa ~/work/test/bbmb.ch $ ls /tmp/bbmbtest/
000000000000.commandLog  000000000015.commandLog  000000000030.commandLog  000000000045.commandLog  000000000060.commandLog  000000000075.commandLog
000000000000.snapshot    000000000015.snapshot    000000000030.snapshot    000000000045.snapshot    000000000060.snapshot    000000000075.snapshot
000000000001.commandLog  000000000016.commandLog  000000000031.commandLog  000000000046.commandLog  000000000061.commandLog  000000000076.commandLog
000000000001.snapshot    000000000016.snapshot    000000000031.snapshot    000000000046.snapshot    000000000061.snapshot    000000000076.snapshot
000000000002.commandLog  000000000017.commandLog  000000000032.commandLog  000000000047.commandLog  000000000062.commandLog  000000000077.commandLog
000000000002.snapshot    000000000017.snapshot    000000000032.snapshot    000000000047.snapshot    000000000062.snapshot    000000000077.snapshot
000000000003.commandLog  000000000018.commandLog  000000000033.commandLog  000000000048.commandLog  000000000063.commandLog  000000000078.commandLog
...

Note

• Testcases saves the data under /tmp/bbmbtest

Experiment

masa@masa /tmp/bbmbtest $ rm *.*
masa@masa /tmp/bbmbtest $ cd ~/work/test
masa@masa ~/work/test $ git clone git://scm.ywesee.com/bbmb.ch
masa@masa ~/work/test/bbmb.ch $ patch -p1 < bbmb_patch
patching file test/test_bbmbapp.rb
patching file test/test_behavior.rb
patching file test/test_order.rb
patching file test/test_reseller.rb
patching file test/test_stock.rb
patching file test/test_user.rb
patching file test/test_validator.rb
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 797 assertions, 12 failures, 28 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 1071 assertions, 3 failures, 0 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 1104 assertions, 0 failures, 0 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 1114 assertions, 0 failures, 0 errors
...

Note

• The first run: 12 failures, 28 errors
• The second run: 3 failures
• Next: no failures but 10 assersions increases per one test/suite.rb run

Next

• Testcases with the updates of new ids process

Testcases with the updates of new ids process

Without the patch

masa@masa /tmp/bbmbtest $ rm *.*
masa@masa ~/work/test $ git clone git://scm.ywesee.com/bbmb.ch
masa@masa ~/work/test $ cd bbmb.ch/
masa@masa ~/work/test/bbmb.ch $ patch -p1 < new_ids_patch
 patching file src/custom/behavior.rb
 patching file src/custom/lookandfeelbase.rb
 patching file src/custom/lookandfeelgag.rb
 patching file src/model/reseller.rb
 patching file src/model/stock.rb
 patching file src/oddb/atcclass.rb
 patching file src/oddb/dose.rb
 patching file src/oddb/fachinfo.rb
 patching file src/oddb/text.rb
 patching file src/oddb/yaml.rb
 patching file src/plugin/yaml.rb
 patching file src/state/result.rb
 patching file src/transactions/stock.rb
 patching file src/util/bbmbapp.rb
 patching file src/util/filehandler.rb
 patching file src/util/session.rb
 patching file src/view/backorders.rb
 patching file src/view/chapter.rb
 patching file src/view/producthistory.rb
 patching file src/view/products.rb
 patching file src/view/result.rb
 patching file src/view/supplier.rb
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb 
187 tests, 757 assertions, 23 failures, 27 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb 
187 tests, 757 assertions, 23 failures, 27 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb 
187 tests, 757 assertions, 23 failures, 27 errors
...

With the patch

masa@masa /tmp/bbmbtest $ rm *.*
masa@masa ~/work/test $ git clone git://scm.ywesee.com/bbmb.ch
masa@masa ~/work/test $ cd bbmb.ch/
masa@masa ~/work/test/bbmb.ch $ patch -p1 < new_ids_patch
 patching file src/custom/behavior.rb
 patching file src/custom/lookandfeelbase.rb
 patching file src/custom/lookandfeelgag.rb
 patching file src/model/reseller.rb
 patching file src/model/stock.rb
 patching file src/oddb/atcclass.rb
 patching file src/oddb/dose.rb
 patching file src/oddb/fachinfo.rb
 patching file src/oddb/text.rb
 patching file src/oddb/yaml.rb
 patching file src/plugin/yaml.rb
 patching file src/state/result.rb
 patching file src/transactions/stock.rb
 patching file src/util/bbmbapp.rb
 patching file src/util/filehandler.rb
 patching file src/util/session.rb
 patching file src/view/backorders.rb
 patching file src/view/chapter.rb
 patching file src/view/producthistory.rb
 patching file src/view/products.rb
 patching file src/view/result.rb
 patching file src/view/supplier.rb
masa@masa ~/work/test/bbmb.ch $ patch -p1 < bbmb_patch 
 patching file test/test_bbmbapp.rb
 patching file test/test_behavior.rb
 patching file test/test_order.rb
 patching file test/test_reseller.rb
 patching file test/test_stock.rb
 patching file test/test_user.rb
 patching file test/test_validator.rb
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 790 assertions, 16 failures, 27 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 840 assertions, 17 failures, 21 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 873 assertions, 14 failures, 21 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 883 assertions, 14 failures, 21 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb
187 tests, 893 assertions, 14 failures, 21 errors

Note

• 14 failures, 21 errors with 10 assertions increasing per test/suite.rb run

Patch

• Attach:new_ids_testcases_patch.20110208.txt

Run

masa@masa /tmp/bbmbtest $ rm *.*
masa@masa ~/work/test $ git clone git://scm.ywesee.com/bbmb.ch
masa@masa ~/work/test $ cd bbmb.ch/
masa@masa ~/work/test/bbmb.ch $ patch -p1 < new_ids_testcases_patch
 patching file src/custom/behavior.rb
 patching file src/custom/lookandfeelbase.rb
 patching file src/custom/lookandfeelgag.rb
 patching file src/model/reseller.rb
 patching file src/model/stock.rb
 patching file src/oddb/atcclass.rb
 patching file src/oddb/dose.rb
 patching file src/oddb/fachinfo.rb
 patching file src/oddb/text.rb
 patching file src/oddb/yaml.rb
 patching file src/plugin/yaml.rb
 patching file src/state/result.rb
 patching file src/transactions/stock.rb
 patching file src/util/bbmbapp.rb
 patching file src/util/filehandler.rb
 patching file src/util/session.rb
 patching file src/view/backorders.rb
 patching file src/view/chapter.rb
 patching file src/view/producthistory.rb
 patching file src/view/products.rb
 patching file src/view/result.rb
 patching file src/view/supplier.rb
 patching file test/test_bbmbapp.rb
 patching file test/test_behavior.rb
 patching file test/test_galdat_export.rb
 patching file test/test_order.rb
 patching file test/test_reseller.rb
 patching file test/test_stock.rb
 patching file test/test_user.rb
 patching file test/test_validator.rb
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb 
184 tests, 838 assertions, 14 failures, 10 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb 
184 tests, 965 assertions, 3 failures, 0 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb 
184 tests, 998 assertions, 0 failures, 0 errors
masa@masa ~/work/test/bbmb.ch $ ruby -W0 test/suite.rb 
184 tests, 1008 assertions, 0 failures, 0 errors

Note

• At the third test run, all failures and errors are gone

Check if run_statistics works

Experiment

ch.bbmb.globopharm> resellers.length
-> 1
ch.bbmb.globopharm> resellers['1']
-> #<BBMB::Reseller:0x7fe073ed11a0>
ch.bbmb.globopharm> resellers['1'].uid
-> 1
ch.bbmb.globopharm> resellers['1'].statistics_mailed.class
-> Array
ch.bbmb.globopharm> resellers['1'].statistics_mailed.length
-> 0
ch.bbmb.globopharm> resellers['1'].mail_statistics(stat_last_month)
-> /usr/lib64/ruby/1.8/net/protocol.rb:206:in `initialize': Connection refused - connect(2)

Note

• failed

Experiment

• src/util/bbmbapp.rb#test_statistics

  def test_statistics
    gid = '1'
    last_month = stat_last_month()
    resellsers[gid].mail_statistics(last_month)
    @prevalence.statistics_mailed(gid, last_month)
  end

Run

• bin/bbmbd
• bin/admin

masa@masa ~/ywesee/bbmb.ch $ bin/admin
ch.bbmb.globopharm> test_statistics

Result

• (still running)

Note

• It is still running and it has taken over 1 hour
• Why does it take so long?

Process

1. Reseller#mail_statistics
2. CustomBehaviorGag#mail_statistics
3. CustomBehaviorGag#make_stats_and_total
   1. HospitalUser#statistics
      1. Order#statistics
4. CustomBehaviorGag#make_statistics_export
5. CustomBehaviorGag#mail_send
6. CustomBehaviorGag#mail_pdf_bill

Experiemnt

ch.bbmb.globopharm> resellers['1'].hospitals.values.inject(0){|sum, hosp| sum+=hosp.all_orders.length}
-> 0

Note

• The number of orders is '0'