view · edit · sidebar · attach · print · history

20100910 import_gkv-error

<< Masa.20100913-test-import_gkv | 2010 | Masa.20100909-job-import_gkv >>


  1. Log import_gkv run last night
  2. Debug ODDB.config
  3. Confirm import_gkv running
  4. Confirm import_dimdi running
  5. Output a log when latest_url is missing
  6. Update import_gkv to set a pdf link directly
  7. Check test/suit.rb de.oddb.org

Goal
  • Resolve import_gkv error / 80%
Milestones
  1. Note log last night import_gkv run 9:15
  2. Debug ODDB.config 10:20
  3. Confirm import_gkv running 11:00
  4. Confirm import_dimdi 13:30 Error / Suspend
  5. Updated setup.de.postgresql.sh 13:30
  6. Error-report import_gkv
    1. Output a log when latest_url is missing 15:30
    2. Update import_gkv to set a pdf link directly 17:00 Suspend
Summary
Commits
ToDo Tomorrow
  1. update import_gkv test case
  2. 20100907 Ticket 242, Comment 9 - notification Mail
  3. Dimdi-Import-Error.
Keep in Mind
Attached Files

Log import_gkv run last night

/usr/lib/ruby/1.8/net/smtp.rb

    def check_auth_args( user, secret, authtype )
p "get-in check_auth_args"
print caller(0).join("\n").to_s,"\n"
      raise ArgumentError, 'both user and secret are required'\
                      unless user and secret
      auth_method = "auth_#{authtype || 'cram_md5'}"
      raise ArgumentError, "wrong auth type #{authtype}"\
                      unless respond_to?(auth_method, true)
    end

Log

"get-in check_auth_args"
E, [2010-09-09T17:27:49.627463 #11283] ERROR -- Gkv: both user and secret are required
/usr/lib64/ruby/1.8/net/smtp.rb:564:in `check_auth_args'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/smtp_tls.rb:8:in `do_start'
/usr/lib64/ruby/1.8/net/smtp.rb:377:in `start'
/usr/lib64/ruby/1.8/net/smtp.rb:315:in `start'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/mail.rb:77:in `sendmail'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/mail.rb:27:in `notify_admins'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:137:in `_reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:102:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:62:in `import_gkv'
/usr/lib64/ruby/1.8/open-uri.rb:32:in `open_uri_original_open'
/usr/lib64/ruby/1.8/open-uri.rb:32:in `open'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/gkv.rb:88:in `download_latest'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:60:in `import_gkv'
import_gkv:12
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `run'
import_gkv:11

Backtrace one by one

/usr/lib64/ruby/1.8/net/smtp.rb:564:in `check_auth_args'

    def check_auth_args( user, secret, authtype )
p "get-in check_auth_args"
print caller(0).join("\n").to_s,"\n"
      raise ArgumentError, 'both user and secret are required'\
                      unless user and secret

This means 'user and secret' is false or nil

http://scm.ywesee.com/?p=de.oddb.org/.git;a=blob;f=lib/oddb/util/smtp_tls.rb;h=39f55594f0d91361d71fd9d66cf0041308ebe015;hb=HEAD#l8/home/masa/ywesee/de.oddb.org/lib/oddb/util/smtp_tls.rb:8:in `do_start'

  def do_start(helodomain, user, secret, authtype)
    raise IOError, 'SMTP session already started' if @started
    check_auth_args user, secret, authtype if user or secret

The user and secret come from user and secret arguments of do_start method.

/usr/lib64/ruby/1.8/net/smtp.rb:377:in `start'

    def start(helo = 'localhost.localdomain',
              user = nil, secret = nil, authtype = nil)   # :yield: smtp
      if block_given?
        begin
          do_start(helo, user, secret, authtype)

The user and secret come from user and secret of arguments of start method.

/usr/lib64/ruby/1.8/net/smtp.rb:315:in `start'

    def SMTP.start(address, port = nil, helo = 'localhost.localdomain',
                   user = nil, secret = nil, authtype = nil,
                   &block)   # :yield: smtp
      new(address, port).start(helo, user, secret, authtype, &block)

The user and secret come from user and secret of argumetns of SMTP.start method.

/home/masa/ywesee/de.oddb.org/lib/oddb/util/mail.rb:77:in `sendmail'

      def Mail.sendmail(mpart, from, recipients)
        config = ODDB.config
        Net::SMTP.start(config.smtp_server, config.smtp_port,
                        config.smtp_domain, config.smtp_user, config.smtp_pass,
                        config.smtp_authtype) do |smtp|

The user and secret come from config.smtp_user and config.smtp_pass. The config == ODDB.config

It means

ODDB.config.smtp_user and ODDB.config.smtp_pass

I should check this ODDB.config

Experiment

/lib/oddb/util/updater.rb

      def Updater.import_gkv(opts = {})
p "get-in Updater.import_gkv"
p ODDB.config.smtp_user
p ODDB.config.smtp_pass
exit

Result

"get-in Updater.import_gkv"
"update@oddb.org"
nil

BINGO!!

This is the cause of the error.

  1. Debug ODDB.config

lib/oddb/config.rb

    'smtp_pass'             => nil,
    'smtp_port'             => 587,
    'smtp_server'           => 'localhost',
    'smtp_user'             => 'update@oddb.org',

This probably means the system did not read the config file, oddb.yml, in my environment.

Experiment

lib/oddb/config.rb

print "ARGV="
p ARGV
  config = RCLConf::RCLConf.new(ARGV, defaults)
  config.load(config.config)
  @config = config
p config.smtp_user
p config.smtp_pass
exit

Result

ARGV=[]
"update@oddb.org"
nil

It looks that config.load does not work.

Experiment

lib/oddb/config.rb

module ODDB
  oddb_dir = File.expand_path('..', ENV['DOCUMENT_ROOT'] || './doc')
  default_dir = File.expand_path('etc', oddb_dir)
  default_config_files = [
    File.join(default_dir, 'oddb.yml'),
    '/etc/oddb/oddb.yml',
  ]
p default_dir
p default_config_files

Result

masa@masa ~/ywesee/de.oddb.org/jobs $ ruby import_gkv 
"/home/masa/ywesee/de.oddb.org/jobs/etc"
["/home/masa/ywesee/de.oddb.org/jobs/etc/oddb.yml", "/etc/oddb/oddb.yml"]
"update@oddb.org"
nil

masa@masa ~/ywesee/de.oddb.org $ ruby jobs/import_gkv 
"/home/masa/ywesee/de.oddb.org/etc"
["/home/masa/ywesee/de.oddb.org/etc/oddb.yml", "/etc/oddb/oddb.yml"]
"mhatakeyama@ywesee.com"
"xxxx"

Summary

  1. We must run import_gkv from de.oddb.org directory
  2. The current directory information is used in the config.rb

Confirm import_gkv running

Result

masa@masa ~/ywesee/de.oddb.org $ ruby jobs/import_gkv
ERROR:  relation "object" already exists
ERROR:  relation "prefetchable_index" already exists
ERROR:  relation "extent_index" already exists
ERROR:  relation "object_connection" already exists
ERROR:  relation "target_id_index" already exists
ERROR:  relation "collection" already exists
ERROR:  relation "target_id_oddb_drugs_galenicform_code" already exists
ERROR:  relation "target_id_oddb_business_company_name" already exists
ERROR:  relation "target_id_oddb_drugs_substance_name" already exists
ERROR:  relation "target_id_oddb_drugs_galenicgroup_name" already exists
ERROR:  relation "target_id_oddb_drugs_package_product" already exists
ERROR:  relation "target_id_oddb_drugs_package_company" already exists
ERROR:  relation "target_id_oddb_drugs_sequence_fachinfo_indications_de" already exists
ERROR:  relation "target_id_oddb_drugs_sequence_product" already exists
ERROR:  relation "target_id_oddb_drugs_substance_code" already exists
ERROR:  relation "target_id_oddb_drugs_package_atc" already exists
ERROR:  relation "target_id_oddb_drugs_substancegroup_name" already exists
ERROR:  relation "target_id_oddb_drugs_unit_name" already exists
ERROR:  relation "target_id_oddb_business_invoice_id" already exists
ERROR:  relation "target_id_oddb_drugs_atc_level_and_code" already exists
ERROR:  relation "target_id_oddb_business_invoice_yus_name" already exists
ERROR:  relation "target_id_oddb_drugs_package_name" already exists
ERROR:  relation "target_id_oddb_text_document_source" already exists
ERROR:  relation "target_id_oddb_drugs_atc_name" already exists
ERROR:  relation "target_id_oddb_drugs_galenicform_description" already exists
ERROR:  relation "target_id_oddb_drugs_sequence_code" already exists
ERROR:  relation "target_id_oddb_drugs_product_name" already exists
ERROR:  relation "target_id_oddb_drugs_package_code" already exists
ERROR:  relation "target_id_oddb_drugs_package_substance" already exists
ERROR:  relation "target_id_oddb_drugs_atc_code" already exists
/home/masa/ywesee/de.oddb.org/lib/oddb/html/view/drugs/package.rb:373: warning: parenthesize argument(s) for future version

-> "220 mx.google.com ESMTP u11sm1513648weq.7\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-STARTTLS\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "STARTTLS\r\n"
-> "220 2.0.0 Ready to start TLS\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-AUTH LOGIN PLAIN XOAUTH\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "AUTH PLAIN AG1oYXRha2V5YW1hQHl3ZXNlZS5jb20AcGFuaW5ldHRh\r\n"
-> "235 2.7.0 Accepted\r\n"
<- "MAIL FROM:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.0 OK u11sm1513648weq.7\r\n"
<- "RCPT TO:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.5 OK u11sm1513648weq.7\r\n"
<- "DATA\r\n"
-> "354  Go ahead u11sm1513648weq.7\r\n"
writing message from String
wrote 679 bytes
-> "250 2.0.0 OK 1284108830 u11sm1513648weq.7\r\n"
<- "QUIT\r\n"
-> "221 2.0.0 closing connection u11sm1513648weq.7\r\n"

It took about 30 minutes.

Report mail

Fre 10 Sep 2010 10:53:47 CEST: de.oddb.org Zubef (PDF)

Fre 10 Sep 2010 10:22:24 CEST: de.oddb.org ODDB::Import::Gkv#import
Imported  6308 Zubef-Entries on 10.09.2010:
Visited   6308 existing Zubef-Entries
Visited   6308 existing Companies
Visited   1000 existing Substances
Created      0 new Zubef-Entries
Created      0 new Products
Created      0 new Sequences
Created      0 new Companies
Created      0 new Substances
Assigned     0 Chemical Equivalences
Assigned     0 Companies
Created      0 Incomplete Packages:

Confirm import_dimdi runnin

It tooks about 20 minuts to run.

Log

masa@masa ~/ywesee/de.oddb.org $ ruby jobs/import_dimdi 
ERROR:  relation "object" already exists
ERROR:  relation "prefetchable_index" already exists
ERROR:  relation "extent_index" already exists
ERROR:  relation "object_connection" already exists
ERROR:  relation "target_id_index" already exists
ERROR:  relation "collection" already exists
ERROR:  relation "target_id_oddb_drugs_galenicform_code" already exists
ERROR:  relation "target_id_oddb_business_company_name" already exists
ERROR:  relation "target_id_oddb_drugs_substance_name" already exists
ERROR:  relation "target_id_oddb_drugs_galenicgroup_name" already exists
ERROR:  relation "target_id_oddb_drugs_package_product" already exists
ERROR:  relation "target_id_oddb_drugs_package_company" already exists
ERROR:  relation "target_id_oddb_drugs_sequence_fachinfo_indications_de" already exists
ERROR:  relation "target_id_oddb_drugs_sequence_product" already exists
ERROR:  relation "target_id_oddb_drugs_substance_code" already exists
ERROR:  relation "target_id_oddb_drugs_package_atc" already exists
ERROR:  relation "target_id_oddb_drugs_substancegroup_name" already exists
ERROR:  relation "target_id_oddb_drugs_unit_name" already exists
ERROR:  relation "target_id_oddb_business_invoice_id" already exists
ERROR:  relation "target_id_oddb_drugs_atc_level_and_code" already exists
ERROR:  relation "target_id_oddb_business_invoice_yus_name" already exists
ERROR:  relation "target_id_oddb_drugs_package_name" already exists
ERROR:  relation "target_id_oddb_text_document_source" already exists
ERROR:  relation "target_id_oddb_drugs_atc_name" already exists
ERROR:  relation "target_id_oddb_drugs_galenicform_description" already exists
ERROR:  relation "target_id_oddb_drugs_sequence_code" already exists
ERROR:  relation "target_id_oddb_drugs_product_name" already exists
ERROR:  relation "target_id_oddb_drugs_package_code" already exists
ERROR:  relation "target_id_oddb_drugs_package_substance" already exists
ERROR:  relation "target_id_oddb_drugs_atc_code" already exists
/home/masa/ywesee/de.oddb.org/lib/oddb/html/view/drugs/package.rb:373: warning: parenthesize argument(s) for future version

-> "220 mx.google.com ESMTP p52sm1536833weq.20\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-STARTTLS\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "STARTTLS\r\n"
-> "220 2.0.0 Ready to start TLS\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-AUTH LOGIN PLAIN XOAUTH\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "AUTH PLAIN AG1oYXRha2V5YW1hQHl3ZXNlZS5jb20AcGFuaW5ldHRh\r\n"
-> "235 2.7.0 Accepted\r\n"
<- "MAIL FROM:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.0 OK p52sm1536833weq.20\r\n"
<- "RCPT TO:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.5 OK p52sm1536833weq.20\r\n"
<- "DATA\r\n"
-> "354  Go ahead p52sm1536833weq.20\r\n"
writing message from String
wrote 464 bytes
-> "250 2.0.0 OK 1284111074 p52sm1536833weq.20\r\n"
<- "QUIT\r\n"
-> "221 2.0.0 closing connection p52sm1536833weq.20\r\n"
-> "220 mx.google.com ESMTP p52sm1536868weq.20\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-STARTTLS\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "STARTTLS\r\n"
-> "220 2.0.0 Ready to start TLS\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-AUTH LOGIN PLAIN XOAUTH\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "AUTH PLAIN AG1oYXRha2V5YW1hQHl3ZXNlZS5jb20AcGFuaW5ldHRh\r\n"
-> "235 2.7.0 Accepted\r\n"
<- "MAIL FROM:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.0 OK p52sm1536868weq.20\r\n"
<- "RCPT TO:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.5 OK p52sm1536868weq.20\r\n"
<- "DATA\r\n"
-> "354  Go ahead p52sm1536868weq.20\r\n"
writing message from String
wrote 2427 bytes
-> "250 2.0.0 OK 1284111078 p52sm1536868weq.20\r\n"
<- "QUIT\r\n"
-> "221 2.0.0 closing connection p52sm1536868weq.20\r\n"
-> "220 mx.google.com ESMTP e31sm2059791wbe.5\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-STARTTLS\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "STARTTLS\r\n"
-> "220 2.0.0 Ready to start TLS\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, [80.219.10.249]\r\n"
-> "250-SIZE 35651584\r\n"
-> "250-8BITMIME\r\n"
-> "250-AUTH LOGIN PLAIN XOAUTH\r\n"
-> "250 ENHANCEDSTATUSCODES\r\n"
<- "AUTH PLAIN AG1oYXRha2V5YW1hQHl3ZXNlZS5jb20AcGFuaW5ldHRh\r\n"
-> "235 2.7.0 Accepted\r\n"
<- "MAIL FROM:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.0 OK e31sm2059791wbe.5\r\n"
<- "RCPT TO:<mhatakeyama@ywesee.com>\r\n"
-> "250 2.1.5 OK e31sm2059791wbe.5\r\n"
<- "DATA\r\n"
-> "354  Go ahead e31sm2059791wbe.5\r\n"
writing message from String
wrote 2817 bytes
-> "250 2.0.0 OK 1284112186 e31sm2059791wbe.5\r\n"
<- "QUIT\r\n"
-> "221 2.0.0 closing connection e31sm2059791wbe.5\r\n"

Report mails

  1. Fre 10 Sep 2010 11:31:12 CEST: de.oddb.org ODDB::Import::Dimdi::Substance (XLS) Good
  2. Fre 10 Sep 2010 11:31:16 CEST: de.oddb.org ODDB::Import::Dimdi::GalenicForm (XLS) Error
  3. Fre 10 Sep 2010 11:49:44 CEST: de.oddb.org Update Festbeträge (XLS) Error

1. Fre 10 Sep 2010 11:31:12 CEST: de.oddb.org ODDB::Import::Dimdi::Substance (XLS)

Fre 10 Sep 2010 11:30:54 CEST: de.oddb.org ODDB::Import::Dimdi::Substance#import
Imported 312 Substances per 01.07.2010:
Visited  271 existing
Visited   86 existing in Combinations
Created    0 new
Created    0 new from Combinations

2. Fre 10 Sep 2010 11:31:16 CEST: de.oddb.org ODDB::Import::Dimdi::GalenicForm (XLS)

Fre 10 Sep 2010 11:31:15 CEST: de.oddb.org ODDB::Import::Dimdi::GalenicForm#import
NoMethodError
private method `split' called for nil:NilClass
/home/masa/ywesee/de.oddb.org/lib/oddb/import/importer.rb:16:in `capitalize_all'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:80:in `import_row'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:31:in `import_worksheet'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `call'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `upto'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/excel/worksheet.rb:34:in `each'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:30:in `import_worksheet'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:25:in `import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:96:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:103:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:103:in `_reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:96:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:27:in `import_dimdi_galenic_forms'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `download'
/usr/lib64/ruby/1.8/open-uri.rb:135:in `open_uri'
/usr/lib64/ruby/1.8/open-uri.rb:518:in `open'
/usr/lib64/ruby/1.8/open-uri.rb:30:in `open'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:39:in `download'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:26:in `import_dimdi_galenic_forms'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:20:in `import_dimdi'
jobs/import_dimdi:12
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `run'
jobs/import_dimdi:11
Imported 216 Galenic Forms per 01.07.2010:
Visited  215 existing
Created    0 new

3. Fre 10 Sep 2010 11:49:44 CEST: de.oddb.org Update Festbeträge (XLS)

Fre 10 Sep 2010 11:31:21 CEST: de.oddb.org ODDB::Import::Dimdi::Product#import
NoMethodError
private method `split' called for nil:NilClass
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:348:in `import_substances'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:281:in `import_sequence'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:258:in `import_product'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:211:in `import_row'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:31:in `import_worksheet'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `call'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:112:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `upto'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/worksheet.rb:111:in `each'
/usr/lib64/ruby/gems/1.8/gems/spreadsheet-0.6.4.1/lib/spreadsheet/excel/worksheet.rb:34:in `each'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:30:in `import_worksheet'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/excel.rb:25:in `import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:96:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:103:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:103:in `_reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:96:in `reported_import'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:34:in `import_dimdi_products'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:40:in `download'
/usr/lib64/ruby/1.8/open-uri.rb:135:in `open_uri'
/usr/lib64/ruby/1.8/open-uri.rb:518:in `open'
/usr/lib64/ruby/1.8/open-uri.rb:30:in `open'
/home/masa/ywesee/de.oddb.org/lib/oddb/import/dimdi.rb:39:in `download'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:33:in `import_dimdi_products'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/updater.rb:21:in `import_dimdi'
jobs/import_dimdi:12
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `call'
/home/masa/ywesee/de.oddb.org/lib/oddb/util/job.rb:16:in `run'
jobs/import_dimdi:11
Imported   30494 Products per 01.07.2010:
Visited      255 existing Products
Visited      148 existing Sequences
Ignored       64 unknown Products
Created      170 new Sequences
Created        0 new Substances from Combinations
Renamed        0 Products
Reassigned     0 PZNs
Deleted        0 Products
Deleted        0 Sequences

After import_gkv and import_dimdi

  • pdf and xls data are saved under var directory

Suspend

Output a log when latest_url is missing

Current problem

  • imoprt_gkv calls Gkv#latest_url and gets the url of PDF source online.
  • But nothing happens (without message) and import_gkv ends, when Gkv#latest_url misses getting the url,
  • Gkv#latest_url is called only from Updater.imoprt_gkv which is called only from jobs/import_gkv script.

Easy solution

  • Output a message on console when Gkv#latest_url misses

Commit

To confirm

  • Do as follows

lib/oddb/import/gkv.rb

  def latest_url agent, opts={}
    host = 'https://www.gkv-spitzenverband.de'
    url = '/Befreiungsliste_Arzneimittel_Versicherte.gkvnet'
    page = agent.get host + url
#    if link = (page/'a[@class=pdf]').first
    if false
      host + link.attributes["href"]
    end
  end

Run import_gkv

masa@masa ~/ywesee/de.oddb.org $ ruby jobs/import_gkv
WARNING: Updater.import_gkv did nothing. It looks failing in grabbing PDF link.
Check HTML source code at https://www.gkv-spitzenverband.de/Befreiungsliste_Arzneimittel_Versicherte.gkvnet
Probably you have to modify Gkv#latest_url method, in particular, this part: link = (page/'a[@class=pdf]')
20100910 masa

Update import_gkv to set a pdf link directly

Goal

  • Add import_gkv option

For example

 $ ruby jobs/import_gkv http://hogehoge/boo.pdf

Plan

  1. Update test cases
  2. lib/oddb/import/gkv.rb#latest_url
  3. lib/oddb/util/updater.rb#Updater.import_gkv
  4. jobs/import_gkv

Commit

Check test/suit.rb de.oddb.org

Just checked it

masa@masa ~/ywesee/de.oddb.org $ ruby test/suite.rb 
/usr/lib64/ruby/1.8/net/http.rb:560:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
        from /usr/lib64/ruby/1.8/net/http.rb:560:in `open'
        from /usr/lib64/ruby/1.8/net/http.rb:560:in `connect'
        from /usr/lib64/ruby/1.8/timeout.rb:53:in `timeout'
        from /usr/lib64/ruby/1.8/timeout.rb:93:in `timeout'
        from /usr/lib64/ruby/1.8/net/http.rb:560:in `connect'
        from /usr/lib64/ruby/1.8/net/http.rb:553:in `do_start'
        from /usr/lib64/ruby/1.8/net/http.rb:542:in `start'
        from /usr/lib64/ruby/1.8/net/http.rb:1035:in `request'
        from /usr/lib64/ruby/1.8/net/http.rb:772:in `get'
        from /usr/lib64/ruby/gems/1.8/gems/Selenium-1.1.14/lib/selenium/openqa/selenium.rb:170:in `do_command'
        from /usr/lib64/ruby/1.8/timeout.rb:62:in `timeout'
        from /usr/lib64/ruby/1.8/timeout.rb:93:in `timeout'
        from /usr/lib64/ruby/gems/1.8/gems/Selenium-1.1.14/lib/selenium/openqa/selenium.rb:158:in `do_command'
        from /usr/lib64/ruby/gems/1.8/gems/Selenium-1.1.14/lib/selenium/openqa/selenium.rb:153:in `stop'
        from ./test/selenium/unit.rb:13
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:32
/usr/lib64/ruby/1.8/net/http.rb:560:in `initialize': Connection refused - connect(2) (Errno::ECONNREFUSED)
        from /usr/lib64/ruby/1.8/net/http.rb:560:in `open'
        from /usr/lib64/ruby/1.8/net/http.rb:560:in `connect'
        from /usr/lib64/ruby/1.8/timeout.rb:53:in `timeout'
        from /usr/lib64/ruby/1.8/timeout.rb:93:in `timeout'
        from /usr/lib64/ruby/1.8/net/http.rb:560:in `connect'
        from /usr/lib64/ruby/1.8/net/http.rb:553:in `do_start'
        from /usr/lib64/ruby/1.8/net/http.rb:542:in `start'
        from /usr/lib64/ruby/1.8/net/http.rb:1035:in `request'
         ... 7 levels...
        from ./test/selenium/unit.rb:49
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `gem_original_require'
        from /usr/lib64/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in `require'
        from test/suite.rb:9

This is probably caused by Selenium library.

@Suspend@

view · edit · sidebar · attach · print · history
Page last modified on July 13, 2011, at 12:03 PM