view · edit · sidebar · attach · print · history

20101026-update-import_gkv debug-import_dimdi

<< Masa.20101027-debug-bsv_follower-autorun | 2010 | Masa.20101025-debug-import_gkv-autorun >>


  1. Update import_gkv option
  2. For README of RCLConf
  3. Debug imoprt_dimdi NoMethodError

Goal
  • update-import_gkv / 100 % done
  • debug import_dimdi / 70 %
Milestones
  1. Test import_gkv option with ODDB.config 9:00
  2. Test import_gkv in local both manually and automatically 10:50
  3. Debug import_dimdi NoMethodError
Summary
Commits
ToDo Tomorrow
  • bsv_followers (oddb.org) autorun bug
Keep in Mind
Attached Files

Update import_gkv option

Experiment

lib/oddb/util/updater.rb

      def Updater.import_gkv(opts = {})
p "getin Updater.import_gkv"
p opts
p ODDB.config
p ODDB.config.test
exit

Run import_gkv

$ jobs/import_gkv test=123

Result

"getin Updater.import_gkv"
{}
#<RCLConf::RCLConf:0x7f178459e000 @defaults={"server_url_export"=>"druby://localhost:11001", "server_name"=>"de.oddb.org", "mail_from"=>"update@oddb.org", "server_url"=>"druby://localhost:11000", "oddb_dir"=>"/home/masa/ywesee/de.oddb.org", "db_backend"=>:psql, "smtp_user"=>"update@oddb.org", "smtp_domain"=>"oddb.org", "session_timeout"=>3600, "export_dir"=>"/home/masa/ywesee/de.oddb.org/var/downloads", "data_dir"=>"/home/masa/ywesee/de.oddb.org/data", "config"=>["/home/masa/ywesee/de.oddb.org/etc/oddb.yml", "/etc/oddb/oddb.yml"], "smtp_port"=>587, "run_updater"=>true, "smtp_authtype"=>:plain, "paypal_server"=>"www.paypal.com", "query_limit_phase"=>86400, "mail_charset"=>"utf8", "http_server"=>"http://localhost", "pharmacy_premium"=>8.1, "export_hour"=>4, "dojo_debug"=>false, "country"=>"DE", "auth_server"=>"drbssl://localhost:9997", "smtp_server"=>"localhost", "mail_invoice_smtp"=>"ipn@oddb.org", "log_file"=>#<IO:0x7f17866a1e78>, "db_user"=>"oddb", "db_name"=>"oddb", "auth_domain"=>"org.oddb.de", "pharmacy_percentage"=>3, "debug_recipients"=>[], "var"=>"/home/masa/ywesee/de.oddb.org/var", "default_dstype"=>"compare", "update_hour"=>2, "smtp_pass"=>nil, "payment_period"=>30, "query_limit"=>5, "mail_invoice_from"=>"\"de.ODDB.org\" ipn@oddb.org", "admins"=>[], "vat"=>19, "remote_databases"=>[], "persistence"=>"odba", "paypal_receiver"=>nil, "log_level"=>"INFO", "db_auth"=>"oddb", "currency_rates"=>"druby://localhost:10999", "run_exporter"=>true, "prices"=>{"org.oddb.de.download.1"=>{"fachinfos.de.oddb.yaml"=>800, "chde.xls"=>600, "compendium_de.oddb.org.htc.prc"=>17, "patinfos_de.oddb.org.firefox.epub"=>17, "patinfos_de.oddb.org.kindle.mobi"=>17, "patinfos.de.oddb.yaml"=>800, "patinfos_de.oddb.org.stanza.epub"=>17, "de.oddb.csv"=>500, "compendium_de.oddb.org.stanza.epub"=>17, "compendium_de.oddb.org.kindle.mobi"=>17, "compendium_de.oddb.org.firefox.epub"=>17, "patinfos_de.oddb.org.htc.prc"=>17, "de.oddb.yaml"=>600}, "org.oddb.de.export.csv"=>0.16, "org.oddb.de.view.30"=>50, "org.oddb.de.view.1"=>5, "org.oddb.de.download.12"=>{"fachinfos.de.oddb.yaml"=>1400, "chde.xls"=>2000, "patinfos.de.oddb.yaml"=>1400, "de.oddb.csv"=>2000, "de.oddb.yaml"=>2500}, "org.oddb.de.view.365"=>400}, "log_level_export"=>"INFO", "log_file_export"=>#<IO:0x7f17866a1e78>, "http_proxies"=>[], "invoice_server"=>"druby://localhost:12375", "download_uncompressed"=>["compendium_de.oddb.org.firefox.epub", "compendium_de.oddb.org.htc.prc", "compendium_de.oddb.org.kindle.mobi", "compendium_de.oddb.org.stanza.epub", "patinfos_de.oddb.org.firefox.epub", "patinfos_de.oddb.org.htc.prc", "patinfos_de.oddb.org.kindle.mobi", "patinfos_de.oddb.org.stanza.epub"], "credentials"=>{}}, @argv={"test"=>"123"}, @runtime={}, @loaded={"smtp_user"=>"mhatakeyama@ywesee.com", "smtp_domain"=>"ywesee.com", "run_updater"=>true, "paypal_server"=>"www.paypal.com", "http_server"=>"http://de.oddb.masa.org", "remote_export_dir"=>"/var/www/oddb.org/data/downloads", "mail_invoice_smtp"=>"zdavatz@ywesee.com", "smtp_server"=>"smtp.gmail.com", "debug_recipients"=>["mhatakeyama@ywesee.com"], "smtp_pass"=>"", "mail_invoice_from"=>"\"Zeno Davatz\" <zdavatz@ywesee.com>", "admins"=>["mhatakeyama@ywesee.com"], "paypal_receiver"=>"zdavatz@ywesee.com", "remote_databases"=>["druby://localhost:10013"], "log_level"=>"DEBUG", "run_exporter"=>true, "credentials"=>{"whocc"=>{"username"=>"hwyss@ywesee.com", "password"=>"a23w33"}, "product_infos"=>{"pop_pass"=>"pharma243298", "pop_user"=>"pharma24@oddb.org", "pop_server"=>"mail.ywesee.com"}}}>
"123"

Run

$ jobs/import_gkv

Result

/usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:42:in `method_missing': private method `test' called for #<RCLConf::RCLConf:0x7f510adc0070> (NoMethodError)
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:42:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:42:in `method_missing'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:41:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:41:in `method_missing'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:40:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:40:in `method_missing'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:39:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:39:in `method_missing'
        from /home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:58:in `import_gkv'
        from jobs/import_gkv:12
        from /home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `call'
        from /home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `run'
        from jobs/import_gkv:11

Notes

Solutions

  1. Modifying the jobs/import_gkv directly
  2. Using ODDB.config (lib/oddb/config)

Which is better...

My opinion

  • The pdf link (url) of import_gkv is not regarding ODDB system itself, so
  • I should not use ODDB.config variable in this case
  • Updating jobs/import_gkv is a straightforward solution and it is easy to understand
  • I guess it is not smart solution, though.

Run import_gkv manually

http://scm.ywesee.com/?p=de.oddb.org/.git;a=blob;f=jobs/import_gkv;h=bcc01f0d2274c7a9c3616f7690dd00410a864288;hb=HEAD»jobs/import_gkv

module ODDB
  module Util
    opt = {}
    if ARGV[0].split("=")[0] == "pdf"
      opt = {:pdf => ARGV[0].split("=")[1]}
    end
    Job.run do
      Updater.import_gkv opt
    end
  end
end

Run import_gkv manually

$ jobs/import_gkv pdf=https://www.gkv-spitzenverband.de/upload/Zuzahlungsbefreit_sort_Name_101015_14671.pdf

Result

Tue Oct 26 08:50:56 2010: de.oddb.org ODDB::Import::Gkv#import
Imported  6557 Zubef-Entries on 26.10.2010:
Visited   6557 existing Zubef-Entries
Visited   6557 existing Companies
Visited   1029 existing Substances
Created      0 new Zubef-Entries
Created      0 new Products
Created      0 new Sequences
Created      0 new Companies
Created      0 new Substances
Assigned    27 Chemical Equivalences
Assigned     3 Companies
Created      0 Incomplete Packages:
Created      1 Product(s) without a name (missing product name):
http://de.oddb.org/de/drugs/product/uid/3480899

Download files

masa@masa ~/ywesee/de.oddb.org $ ls var/pdf/gkv/
2010.10.26-Zuzahlungsbefreit_sort_Name_101015_14671.pdf  Zuzahlungsbefreit_sort_Name_101015_14671.pdf

Run import_gkv automatically

http://scm.ywesee.com/?p=de.oddb.org/.git;a=blob;f=lib/oddb/util/server.rb;h=e39369992f80402a87e7b38807dd52d783f74789;hb=HEAD»lib/oddb/util/server.rb

      def initialize(*args)
        super
        @rss_mutex = Mutex.new
#        run_exporter if(ODDB.config.run_exporter)
        run_updater if(ODDB.config.run_updater)
      end


      def run_at(hour, &block)
        Thread.new {
          loop {
            now = Time.now
            #run_at = Time.local(now.year, now.month, now.day, hour)
            run_at = Time.local(now.year, now.month, now.day, now.hour, 45)
            while(now > run_at)
              run_at += 24*60*60
            end
print "now=", now, "\n"
print "run_at=", run_at, "\n"
            sleep(run_at - now)
            block.call
          }
        }
      end

Run oddbd (Run import_gkv automatically)

masa@masa ~/ywesee/de.oddb.org $ bin/oddbd
now=Tue Oct 26 09:40:33 +0200 2010
run_at=Tue Oct 26 09:45:00 +0200 2010
I, [2010-10-26T09:40:33.598382 #25828]  INFO -- start: starting oddb-server on druby://localhost:11000

Result

  • Emails
  1. Tue Oct 26 09:45:20 2010: de.oddb.org ODDB::Import::Dimdi::Substance (XLS) success
  2. Tue Oct 26 09:45:24 2010: de.oddb.org ODDB::Import::Dimdi::GalenicForm (XLS) NoMthodError
  3. Tue Oct 26 10:05:04 2010: de.oddb.org Update Festbeträge (XLS) NoMethodError
  4. Tue Oct 26 10:38:59 2010: de.oddb.org Zubef (PDF) success

Check new files

masa@masa ~/ywesee/de.oddb.org $ git status
# On branch work
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working directory)
#
#       modified:   jobs/import_gkv
#       modified:   lib/oddb/util/server.rb
#
# Untracked files:
#   (use "git add <file>..." to include in what will be committed)
#
#       data/sql/
#       etc/
#       gitignore
#       log/
#       var/
no changes added to commit (use "git add" and/or "git commit -a")

masa@masa ~/ywesee/de.oddb.org $ ls log -al
insgesamt 8
drwxr-xr-x  2 masa masa   16 26. Okt 10:05 .
drwxr-xr-x 12 masa masa 4096 26. Okt 09:45 ..
-rw-r--r--  1 masa masa    0 26. Okt 09:45 import_dimdi
-rw-r--r--  1 masa masa    0 26. Okt 10:05 import_gkv
masa@masa ~/ywesee/de.oddb.org $ ls var/
pdf  xls
masa@masa ~/ywesee/de.oddb.org $ ls var/pdf/
gkv
masa@masa ~/ywesee/de.oddb.org $ ls var/pdf/gkv/
2010.10.26-Zuzahlungsbefreit_sort_Name_101015_14671.pdf  Zuzahlungsbefreit_sort_Name_101015_14671.pdf
masa@masa ~/ywesee/de.oddb.org $ ls var/xls/
darreichungsformen-201007.xls  festbetraege-201007.xls  wirkstoffkuerzel-201007.xls

Commit Updated import_gkv pdf link option

For README of RCLConf

This is used at de.oddb.org/lib/oddb/config.rb

Summary

  • This library is available and useful for a configuration of an application.
  • A command line option is also managed in this library.

For example

config.rb

require 'rclconf'

module Application
default_config_files = [
  '/home/masa/work/config.yml'
]
defaults = {
  'admins'  => [],
  'passwd'  => nil,
}

@config = RCLConf::RCLConf.new(ARGV, defaults)
@config.load(default_config_files)
end

config.yml

---
admins:
 - masa@ywesee.com
users:
 - abc@ywesee.com
 - xyw@ywesee.com
passwd: 'hata'

application.rb

require 'config'

module Application
print "@config="
p @config
print "@config.admins="
p @config.admins
print "@config.users="
p @config.users
print "@config.passwd="
p @config.passwd
print "@config.parse_argv(ARGV)="
p @config.parse_argv(ARGV)
print "@config.test="
p @config.test
end

Run application (with a new command line argument)

$ ruby application.rb test=123
@config=#<RCLConf::RCLConf:0x7f9e3b0696c8 @defaults={"passwd"=>nil, "admins"=>[]}, @argv={"test"=>"123"}, @runtime={}, @loaded={"passwd"=>"hata", "users"=>["abc@ywesee.com", "xyw@ywesee.com"], "admins"=>["masa@ywesee.com"]}>
@config.admins=["masa@ywesee.com"]
@config.users=["abc@ywesee.com", "xyw@ywesee.com"]
@config.passwd="hata"
@config.parse_argv(ARGV)={"test"=>"123"}
@config.test="123"

Run application (without a command line arugemnt)

$ ruby application.rb
@config=#<RCLConf::RCLConf:0x7f6ddae6f448 @defaults={"passwd"=>nil, "admins"=>[]}, @argv={}, @runtime={}, @loaded={"passwd"=>"hata", "users"=>["abc@ywesee.com", "xyw@ywesee.com"], "admins"=>["masa@ywesee.com"]}>
@config.admins=["masa@ywesee.com"]
@config.users=["abc@ywesee.com", "xyw@ywesee.com"]
@config.passwd="hata"
@config.parse_argv(ARGV)={}
/usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:42:in `method_missing': private method `test' called for #<RCLConf::RCLConf:0x7f6ddae6f448> (NoMethodError)
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:42:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:42:in `method_missing'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:41:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:41:in `method_missing'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:40:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:40:in `method_missing'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:39:in `fetch'
        from /usr/lib64/ruby/site_ruby/1.8/rclconf/rclconf.rb:39:in `method_missing'
        from application.rb:16
@config.test=

Notes

  • @config variable can change dynamically
  • but NoMethodError comes when an undefined configuration option is implemented (this should be updated)

Check tests

masa@masa ~/ywesee/rclconf $ ruby test/test_rclconf.rb 
Loaded suite test/test_rclconf
Started
...E.EE.E......
Finished in 0.003053 seconds.

  1) Error:
test_argv_overrides_file(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:48:in `test_argv_overrides_file'
    test/test_rclconf.rb:47:in `open'
    test/test_rclconf.rb:47:in `test_argv_overrides_file'

  2) Error:
test_file_overrides_defaults(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:60:in `test_file_overrides_defaults'
    test/test_rclconf.rb:59:in `open'
    test/test_rclconf.rb:59:in `test_file_overrides_defaults'

  3) Error:
test_load_mandatory__file_found(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:80:in `test_load_mandatory__file_found'
    test/test_rclconf.rb:79:in `open'
    test/test_rclconf.rb:79:in `test_load_mandatory__file_found'

  4) Error:
test_load_optional__file_precedence(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:107:in `test_load_optional__file_precedence'
    test/test_rclconf.rb:106:in `open'
    test/test_rclconf.rb:106:in `test_load_optional__file_precedence'

15 tests, 18 assertions, 0 failures, 4 errors

Check a test case which succeeds

  • No such a test case, all tests failed
masa@masa ~/work/rclconf $ git checkout 51b47ccbb9aa994e50289434ff87c9889de1c16a
masa@masa ~/work/rclconf $ ruby test/test_rclconf.rb 
Loaded suite test/test_rclconf
Started
..E.EFE.FE....
Finished in 0.004718 seconds.

  1) Error:
test_argv_overrides_file(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:39:in `test_argv_overrides_file'
    test/test_rclconf.rb:38:in `open'
    test/test_rclconf.rb:38:in `test_argv_overrides_file'

  2) Error:
test_file_overrides_defaults(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:52:in `test_file_overrides_defaults'
    test/test_rclconf.rb:51:in `open'
    test/test_rclconf.rb:51:in `test_file_overrides_defaults'

  3) Failure:
test_load_mandatory__corrupt_file__raises_error(RCLConf::RCLConfTest) [test/test_rclconf.rb:72]:
<ArgumentError> exception expected but none was thrown.

  4) Error:
test_load_mandatory__file_found(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:84:in `test_load_mandatory__file_found'
    test/test_rclconf.rb:83:in `open'
    test/test_rclconf.rb:83:in `test_load_mandatory__file_found'

  5) Failure:
test_load_optional__corrupt_file__raises_error(RCLConf::RCLConfTest) [test/test_rclconf.rb:106]:
<ArgumentError> exception expected but none was thrown.

  6) Error:
test_load_optional__file_precedence(RCLConf::RCLConfTest):
NoMethodError: undefined method `>' for nil:NilClass
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
    /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
    /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
    /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
    test/test_rclconf.rb:123:in `test_load_optional__file_precedence'
    test/test_rclconf.rb:122:in `open'
    test/test_rclconf.rb:122:in `test_load_optional__file_precedence'

14 tests, 13 assertions, 2 failures, 4 errors

Notes

  • I guess this is not from rclconf but from yaml library
  • This might be related to YDPM error in my local.

Experiment

require 'yaml'
YAML.dump( ['badger', 'elephant', 'tiger'], File.open('sample.yml', 'w'))

Result

$ ruby test.rb 
/usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?': undefined method `>' for nil:NilClass (NoMethodError)
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:110:in `node_export'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:110:in `add'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:110:in `to_yaml'
         ... 6 levels...
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:107:in `to_yaml'
        from /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
        from test2.rb:3

masa@masa ~/work $ ruby1.9 test2.rb 
test2.rb:in `require': no such file to load -- auto_gem (LoadError)

masa@masa ~/work $ unset RUBYOPT
masa@masa ~/work $ ruby1.9 test2.rb 
masa@masa ~/work $ cat sample.yml 
--- 
- badger
- elephant
- tiger

Test

masa@masa ~/work/rclconf $ unset RUBYOPT
masa@masa ~/work/rclconf $ ruby1.9 test/test_rclconf.rb 
Loaded suite test/test_rclconf
Started
...............
Finished in 0.002902 seconds.

15 tests, 25 assertions, 0 failures, 0 errors, 0 skips

Test run options: --seed 53894

Conclusion

  • Ruby 1.8.6 causes this error

Memo

  • I can run correctly YDPM in my local as follows: 'ruby1.9 -I /home/masa/ywesee/rclconf/lib /usr/bin/ydpm'

Debug imoprt_dimdi NoMethodError

There comes NoMethodError mails when import_dimdi runs

  1. Tue Oct 26 09:45:24 2010: de.oddb.org ODDB::Import::Dimdi::GalenicForm (XLS) NoMthodError
  2. Tue Oct 26 10:05:04 2010: de.oddb.org Update Festbeträge (XLS) NoMethodError
  3. Tue Oct 26 09:45:20 2010: de.oddb.org ODDB::Import::Dimdi::Substance (XLS) success

Details

1. Tue Oct 26 09:45:24 2010: de.oddb.org ODDB::Import::Dimdi::GalenicForm (XLS) NoMthodError

Tue Oct 26 09:45:23 2010: de.oddb.org ODDB::Import::Dimdi::GalenicForm#import
NoMethodError
private method `split' called for nil:NilClass
/home/masa/ywesee/de.oddb.org/lib/oddb/import/importer.rb:16:in `capitalize_all'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:80:in `import_row'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:31:in `import_worksheet'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `call'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `upto'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/excel/worksheet.rb:34:in `each'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:30:in `import_worksheet'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:25:in `import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:110:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:117:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:117:in `_reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:110:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:27:in `import_dimdi_galenic_forms'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `download'
/usr/lib64/ruby/1.8/open-uri.rb:135:in `open_uri'
/usr/lib64/ruby/1.8/open-uri.rb:518:in `open'
/usr/lib64/ruby/1.8/open-uri.rb:30:in `open'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:39:in `download'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:26:in `import_dimdi_galenic_forms'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:20:in `import_dimdi'
/home/masa/ywesee/de.oddb.org/jobs/import_dimdi:12
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `run'
/home/masa/ywesee/de.oddb.org/jobs/import_dimdi:11
Imported 216 Galenic Forms per 01.07.2010:
Visited  215 existing
Created    0 new

2. Tue Oct 26 10:05:04 2010: de.oddb.org Update Festbeträge (XLS) NoMethodError

Tue Oct 26 09:45:29 2010: de.oddb.org ODDB::Import::Dimdi::Product#import
NoMethodError
private method `split' called for nil:NilClass
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:348:in `import_substances'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:281:in `import_sequence'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:258:in `import_product'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:211:in `import_row'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:31:in `import_worksheet'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `call'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `upto'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/excel/worksheet.rb:34:in `each'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:30:in `import_worksheet'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:25:in `import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:110:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:117:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:117:in `_reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:110:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:34:in `import_dimdi_products'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `download'
/usr/lib64/ruby/1.8/open-uri.rb:135:in `open_uri'
/usr/lib64/ruby/1.8/open-uri.rb:518:in `open'
/usr/lib64/ruby/1.8/open-uri.rb:30:in `open'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:39:in `download'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:33:in `import_dimdi_products'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:21:in `import_dimdi'
/home/masa/ywesee/de.oddb.org/jobs/import_dimdi:12
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `run'
/home/masa/ywesee/de.oddb.org/jobs/import_dimdi:11
Imported   30494 Products per 01.07.2010:
Visited        0 existing Products
Visited        0 existing Sequences
Ignored        1 unknown Products
Created        0 new Sequences
Created        0 new Substances from Combinations
Renamed        0 Products
Reassigned     0 PZNs
Deleted        0 Products
Deleted        0 Sequences

Notes

First focusing on 1. Tue Oct 26 09:45:24 2010: de.oddb.org ODDB::Import::Dimdi::GalenicForm (XLS)

lib/oddb/util/updater.rb:20

      def Updater.import_dimdi
        if(date = Import::Dimdi.current_date(DIMDI_INDEX))
          import_dimdi_substances(date)
          import_dimdi_galenic_forms(date)  #<= here
          import_dimdi_products(date)
        end
      end

lib/oddb/util/updater.rb:26

      def Updater.import_dimdi_galenic_forms(date)
        file = date.strftime("darreichungsformen-%Y%m.xls")
        Import::Dimdi.download(file) { |io|                        #<= here
          reported_import(Import::Dimdi::GalenicForm.new(date), io,
                          :filetype => 'XLS')
        }
      end

lib/oddb/import/dimdi.rb:39

  def Dimdi.download(file, &block)
    url = File.join(DIMDI_PATH, file)
    xls_dir = File.join(ODDB.config.var, 'xls')
    FileUtils.mkdir_p(xls_dir)
    dest = File.join(xls_dir, file)
    unless(File.exist?(dest))
      open(url) { |remote|         #<= here
        block.call(remote)
        remote.rewind
        open(dest, 'w') { |local|
          local << remote.read
        }
      }
    end
  rescue StandardError => e
    ODDB.logger.error('Dimdi') { e.message }
  end

skip

lib/oddb/import/dimdi.rb:80

    def import_row(row)
      @count += 1
      abbr = cell(row, 0)
      description = capitalize_all(cell(row, 1))                   #<= here
      galenic_form = Drugs::GalenicForm.find_by_code(:value => abbr,
                                                     :type => "galenic_form",
                                                     :country => 'DE')
      galenic_form ||= Drugs::GalenicForm.find_by_description(description)
      if(galenic_form)
        @existing += 1
      else
        @created += 1
        galenic_form = Drugs::GalenicForm.new
        galenic_form.description.de = description
      end
      galenic_form.add_code(Util::Code.new("galenic_form",
                                           abbr, 'DE', @date))
      galenic_form.save
      galenic_form
    end

lib/oddb/import/importer.rb:16

      def capitalize_all(str)
        ## benchmarked fastest against an append (<<) solution
        str.split(/\b/).collect { |part|                           #<= here
          @@lower.match(part) ? part.downcase : part.capitalize }.join
      end

Experiment

lib/oddb/import/importer.rb:16

      def capitalize_all(str)
p str
        ## benchmarked fastest against an append (<<) solution
        str.split(/\b/).collect { |part|                           #<= here
          @@lower.match(part) ? part.downcase : part.capitalize }.join
      end

Result

...
u"ÜBERZOGENE TABLETTEN"
u"ÜBERZOGENE TABLETTEN"
u"VAGINALCREME"
u"VAGINALDEPOTOVULUM"
u"VAGINALOVULA"
u"VAGINALTABLETTEN"
u"VAGINALZAEPFCHEN (SUPPOS.)"
u"VAGINALSALBE"
u"WEICHGELATINEKAPSELN"
u"ZERBEISSKAPSELN"
nil

Notes

  • I have confirmed that it certainly became nil

Backtrace again

lib/oddb/import/excel.rb:31

      def import_worksheet(worksheet)
        worksheet.each(@skip_rows) { |row|
          import_row(row)                  #<= here
        }
      end

Experiment

lib/oddb/import/excel.rb:31

      def import_worksheet(worksheet)
print "worksheet="
p worksheet
        worksheet.each(@skip_rows) { |row|
          import_row(row)
        }
      end
      def parse(io)
        Spreadsheet.open(io)
      end
    end

Result

worksheet=#<Spreadsheet::Excel::Worksheet:0x003f85ad39a644 @changes= @columns= @reader=#<Spreadsheet::Excel::Reader:0x7f0b5a7bc4f8> @row_addresses= @selected= @ole=#<Ole::Storage::RangesIOMigrateable:0x7f0b5a7ae4e8> @name=Darreichungsform 01.07.2010 @links= @default_format= @offsets= @offset=8863 @workbook=#<Spreadsheet::Excel::Workbook:0x7f0b5a7b0450> @dimensions= @rows[397]>

Check xls file

  • var/xls/darreichungsformen-201007.xls

Notes

  • The data certainly ends at 'ZERBEISSKAPSELN' but the script tried to read the next data row
  • Then the error happened

Question

  • Why does not the script end at the end of data in the xls file?

Experiment

lib/oddb/import/dimdi.rb

      abbr = cell(row, 0)
      value = cell(row, 1)
print "abbr="
p abbr
print "value="
p value
print "abbr && value="
p (abbr && value)
      if(abbr != nil && value != nil)
      @count += 1
      description = capitalize_all(cell(row, 1))
      galenic_form = Drugs::GalenicForm.find_by_code(:value => abbr,
                                                     :type => "galenic_form",
                                                     :country => 'DE')
      galenic_form ||= Drugs::GalenicForm.find_by_description(description)
      if(galenic_form)
        @existing += 1
      else
        @created += 1
        galenic_form = Drugs::GalenicForm.new
        galenic_form.description.de = description
      end
      galenic_form.add_code(Util::Code.new("galenic_form",
                                           abbr, 'DE', @date))
      galenic_form.save
      galenic_form
      end

Result

Tue Oct 26 14:41:11 2010: de.oddb.org ODDB::Import::Dimdi::GalenicForm#import
Imported 215 Galenic Forms per 01.07.2010:
Visited  215 existing
Created    0 new

Commit Debug import_dimdi_galenic_forms nil data error in a xls file (NoMethodError)

Next focusing on 2. Tue Oct 26 10:05:04 2010: de.oddb.org Update Festbeträge (XLS) NoMethodError

Confirm the error

lib/oddb/util/updater.rb

      def Updater.import_dimdi
        if(date = Import::Dimdi.current_date(DIMDI_INDEX))
#          import_dimdi_substances(date)
#          import_dimdi_galenic_forms(date)
          import_dimdi_products(date)
        end
      end

Run jobs/import_dimdi

Result

Tue Oct 26 15:11:42 2010: de.oddb.org ODDB::Import::Dimdi::Product#import
NoMethodError
private method `split' called for nil:NilClass
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:351:in `import_substances'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:284:in `import_sequence'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:261:in `import_product'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:214:in `import_row'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:31:in `import_worksheet'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `call'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `upto'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/excel/worksheet.rb:34:in `each'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:30:in `import_worksheet'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:25:in `import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:110:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:117:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:117:in `_reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:110:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:34:in `import_dimdi_products'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `download'
/usr/lib64/ruby/1.8/open-uri.rb:135:in `open_uri'
/usr/lib64/ruby/1.8/open-uri.rb:518:in `open'
/usr/lib64/ruby/1.8/open-uri.rb:30:in `open'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:39:in `download'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:33:in `import_dimdi_products'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:21:in `import_dimdi'
jobs/import_dimdi:12
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `run'
jobs/import_dimdi:11
Imported   30494 Products per 01.07.2010:
Visited        0 existing Products
Visited        0 existing Sequences
Ignored        1 unknown Products
Created        0 new Sequences
Created        0 new Substances from Combinations
Renamed        0 Products
Reassigned     0 PZNs
Deleted        0 Products
Deleted        0 Sequences

New files

masa@masa ~/ywesee/de.oddb.org $ ls var
xls
masa@masa ~/ywesee/de.oddb.org $ ls var/xls/
festbetraege-201007.xls

Backtrace the error message

lib/oddb/import/dimdi.rb:351

      if(subs.empty?)
        names = groupname.split('+')         #<= here
        subs = names.collect { |name|
          assumed_name = name.strip[/^\S+/]
          unless(assumed_name.empty?)
            sub = Drugs::Substance.find_by_name(assumed_name)
            if(sub.nil?)
              @created_substances += 1
              sub = Drugs::Substance.new
              sub.name.de = assumed_name
              sub.save
            end
            if(names.size == 1)
              assign_substance_group(sub, groupname)
            end
            sub
          end
        }
      end
      subs.compact
    end

Experiment

lib/oddb/import/dimdi.rb

    def import_row(row)
      @package_date = cell(row, 13) || @date
      @count += 1
      pzn = u(cell(row, 0).to_i.to_s)
p cell(row, 0)
p pzn

Run jobs/import_dimdi

Result

u"9194307"
9194313.0
u"9194313"
9194336.0
u"9194336"
nil
u"0"

Look at the xls file

Notes

  • This is also caused by nil data in a xls file.

Experiment

lib/oddb/import/dimdi.rb

    def import_row(row)
      if(value = cell(row, 0))
        @package_date = cell(row, 13) || @date
        @count += 1
        pzn = u(value.to_i.to_s)
        package = Drugs::Package.find_by_code(:type    => 'cid',
                                            :value   => pzn,
                                            :country => 'DE')
        name = product_name(row)
        product = Drugs::Product.find_by_name(name)
        if !package
          ## new package and possibly new product
          import_product row, product, name
        elsif !package.product
          product ||= create_product name
          if package.sequence.nil?
            import_sequence row, product, package
          else
            seq = package.sequence
            seq.product = product
            seq.save
            update_package row, package
          end
        else
          ## update package-data
          update_package row, package
        end
      end
    end

Run jobs/import_dimdi

Result

Tue Oct 26 16:21:46 2010: de.oddb.org ODDB::Import::Dimdi::Product#import
Imported   30493 Products per 01.07.2010:
Visited        0 existing Products
Visited        0 existing Sequences
Ignored        0 unknown Products
Created        0 new Sequences
Created        0 new Substances from Combinations
Renamed        0 Products
Reassigned     0 PZNs
Deleted        0 Products
Deleted        0 Sequences

Commit Debug import_product nil data error in a xls file (NoMethodError)

view · edit · sidebar · attach · print · history
Page last modified on July 13, 2011, at 11:57 AM