view · edit · sidebar · attach · print · history

20101221-update-ch_oddb_org-generics_xls

<< Masa.20101222-update-ch_oddb_org-generics_xls | 2010 | Masa.20101220-debug-ch_oddb_org-generics_xls >>

Confirm the last night run of export_generics
Implement both reporting and skipping
testenvironment.rb
Check again with the latest data today

Goal

Update export_generics / 90%

Milestones

~~Confirm the last night run of export_generics~~ 7:45
~~Update generics_xls~~
- ~~Implement of a function of both reporting an email and skiping nil data~~ 13:45
testenvironment.rb updating
Check again with the latest data
Test cases (there is no test cases for export_generics method)
- Package#comparable?
- export_generics

Summary

http://masay.ch/2010/12/21/update-ch_oddb_org-generics_xls/

Commits

Skip nil basename packages when export_generics (without reporting) (oddb.org)
Debug: Skip nil basename packages and report them during export_generics (oddb.org)
Added requirement of testenvironment2 for tests in local environment (oddb.org)

ToDo Tomorrow
Keep in Mind

swissmedic_followers debug
rpdf2txt announcement 20101214
different page 2010, 2011 and navigator link at the bottom of page PM-Wiki Markup for paging between pages
On Ice
export_fachinfo test locally (weekend)
backup script (weekend)
emerge --sync

Confirm the last night run of export_generics

Email 17:00 run, 18:35 done. ch.ODDB.org Report - Generikaliste - 12/2010

/home/masa/ywesee/oddb.org/data/downloads/generics.xls
(There is an attached file (generics.xls))

Commit

Skip nil basename packages when export_generics (without reporting) (oddb.org)

Implement a function of reporting with skipping nil data

Memo

The current updating takes too much time
The following code takes long time to calculate

pac.comparables.select{|pack| pack.basename.nil?}.empty?

Experiment

ext/export/src/generics_xls.rb#export_generics

            def export_generics
                originals = []
                generics = []
                comparables = []
                @app.each_package { |pac|
if pac.basename.nil?
open("/home/masa/work/test.dat","a"){|o| o.print "pac.basename.nil?=", pac.barcode, "\n"}
end
                    #if(pac.public? && pac.registration.active? && !pac.basename.nil? && pac.comparables.select{|pack| pack.basename.nil?}.empty?)
                    if(pac.public? && pac.registration.active? && !pac.basename.nil?)

src/model/package.rb#comparable?

    def comparable?(bottom, top, pack)
      begin
        pack != self \
          && (other = pack.comparable_size) \
          && bottom < other \
          && top > other \
&& !pack.basename.nil?
      rescue RuntimeError => e
        false
      end
    end

Reboot
1. ext/export/bin/exportd
2. bin/oddbd
Run Updater.new(self).export_generics_xls from bin/admin

Result

It takes a few minutes to run export_generics
Email ch.ODDB.org Report - Generikaliste - 12/2010 Success

masa@masa ~/ywesee/oddb.org $ cat ~/work/test.dat 
pac.basename.nil?=7680262930302
pac.basename.nil?=7680262930494
pac.basename.nil?=7680179810568
pac.basename.nil?=7680179810483
pac.basename.nil?=7680570540019
pac.basename.nil?=7680570540033
pac.basename.nil?=7680557580014
pac.basename.nil?=7680556540071
pac.basename.nil?=7680556540095
pac.basename.nil?=7680556540019
pac.basename.nil?=7680556540033
pac.basename.nil?=7680556540057

Notes

The nil packages increased from yesterday?
Let's check it again

Experiment

ext/export/src/generics_xls.rb#export_generics

            def export_generics
                originals = []
                generics = []
                comparables = []
                @app.each_package { |pac|
if pac.basename.nil?
open("/home/masa/work/test.dat","a"){|o| o.print "pac.basename.nil?=", pac.barcode, "\n"}
end
                    #if(pac.public? && pac.registration.active? && !pac.basename.nil? && pac.comparables.select{|pack| pack.basename.nil?}.empty?)
                    #if(pac.public? && pac.registration.active? && !pac.basename.nil?)
                    if(pac.public? && pac.registration.active?)

src/model/package.rb#comparable?

    def comparable?(bottom, top, pack)
      begin
        pack != self \
          && (other = pack.comparable_size) \
          && bottom < other \
          && top > other 
#&& !pack.basename.nil?
      rescue RuntimeError => e
        false
      end
    end

Reboot
1. ext/export/bin/exportd
2. bin/oddbd
Run Updater.new(self).export_generics_xls from bin/admin

Result

Email ch.ODDB.org Report - Error: Generikaliste - 12/2010

Plugin: ODDB::XlsExportPlugin
Error: StandardError
Message: Package basename is nil. The package is not comparable.

Package (company, EAN code): Merx Pharma (Schweiz) AG, 7680570540019, Merx Pharma (Schweiz) AG, 7680570540033

masa@masa ~/ywesee/oddb.org $ cat ~/work/test.dat 
mhatakeyama@ywesee.com
pac.basename.nil?=7680262930302
pac.basename.nil?=7680262930494
pac.basename.nil?=7680179810568
pac.basename.nil?=7680179810483
pac.basename.nil?=7680570540019
pac.basename.nil?=7680570540033
pac.basename.nil?=7680557580014
pac.basename.nil?=7680556540071
pac.basename.nil?=7680556540095
pac.basename.nil?=7680556540019
pac.basename.nil?=7680556540033
pac.basename.nil?=7680556540057

Notes

It is the same as the last one, namely these are all nil basename packages

Experiment

ext/export/src/generics_xls.rb#export_generics

require 'util/log'
....
            def export_generics
                originals = []
                generics = []
                comparables = []
                @app.each_package { |pac|
                    #if(pac.public? && pac.registration.active? && !pac.basename.nil? && pac.comparables.select{|pack| pack.basename.nil?}.empty?)
                    #if(pac.public? && pac.registration.active? && !pac.basename.nil?)
                    if(pac.public? && pac.registration.active?)
                        if(pac.registration.original? && (comps = pac.comparables) \
                             && !comps.empty?)
                            originals.push(pac)
                            comparables.concat(comps)
                        elsif(pac.registration.generic?)
                            generics.push(pac)
                        end
                    end
                }
                # Check Packages
                # Some packages cannot be compared if package basename is nil
                #nilpackages = originals.select{|pac| pac.basename.nil?}.map{|pac| [pac.company_name, pac.barcode]}
                nilpackages = originals.select{|pac| pac.basename.nil?}
                unless nilpackages.empty?
                  message =  "Package basename is nil.\n"
                  message << "The following packages are skipped during export_generics, since the packages are not comparable.\n\n"
                  link = "http://ch.oddb.org/de/gcc/search/zone/drugs/search_query/"
                  message << "Package (company, EAN code, link):\n"
                  message << nilpackages.map{|pac| [pac.company_name, pac.barcode, link + pac.barcode].join(", ")}.join("\n")
                  log = Log.new(@@today)
                  log.report = [
                    "Class: #{self.class}",
                    "Method: export_generics",
                    "Message: #{message}",
                    "Backtrace:",
                    caller(0).join("\n"),
                  ].join("\n")
                  log.recipients = RECIPIENTS.dup
                  log.notify("Warning: Nil basename packages")

                  originals -= nilpackages
                end

                originals.sort.each { |pac| export_comparables(pac) }
                @rows += 1 # leave a space in the xls
                (generics - comparables).sort.each { |pac| export_generic(pac) }
                @rows
            end

src/model/package.rb#comparable?

    def comparable?(bottom, top, pack)
      begin
        pack != self \
          && (other = pack.comparable_size) \
          && bottom < other \
          && top > other \
&& !pack.basename.nil?
      rescue RuntimeError => e
        false
      end
    end

Reboot
1. ext/export/bin/exportd
2. bin/oddbd
Run Updater.new(self).export_generics_xls from bin/admin

Result

EMail
- ch.ODDB.org Report - Generikaliste - 12/2010 success exporting
- ch.ODDB.org Report - Warning: Nil basename packages - 12/2010

Class: ODDB::OdbaExporter::GenericXls
Method: export_generics
Message: Package name may have a number in it and therefore basename becomes nil.
The following packages are skipped during export_generics, since the packages are not comparable.

Package (company, EAN code, link):
Merx Pharma (Schweiz) AG, 7680570540019, http://ch.oddb.org/de/gcc/search/zone/drugs/search_query/7680570540019
Merx Pharma (Schweiz) AG, 7680570540033, http://ch.oddb.org/de/gcc/search/zone/drugs/search_query/7680570540033

Backtrace:
/home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:122:in `export_generics'
/home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:98:in `export_generics_xls'
/home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:255:in `call'
/home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:255:in `safe_export'
/usr/lib64/ruby/1.8/tempfile.rb:172:in `open'
/home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:254:in `safe_export'
/home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:96:in `export_generics_xls'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1516:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1590:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1431:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1348:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `start_service'
bin/exportd:14

Notes

'basename' method is defined in src/model/sequence.rb

@name_base.to_s[/^.[^0-9]+/u]

This returns nil if the name has a number at the beginning of the name (@name_base)
I am not sure this is a bug or design

Commit

Debug: Skip nil basename packages and report them during export_generics (oddb.org)

testenvironment.rb

Memo

Usually, the report mail will send 'mhatakeyama' and 'zdavatz'.
But in the tests locally, it should be only to 'mhatakeyama'

Experiment

ext/export/src/generics_xls.rb#export_generics

            def export_generics
open("/home/masa/work/test.dat","w"){|o| o.print RECIPIENTS, "\n"}
exit
...

begin
      require 'testenvironment2'
rescue LoadError
end

src/testenvironment2.rb

...
  module OdbaExporter
   class GenericXls
     RECIPIENTS = ['mhatakeyama@ywesee.com']
   end
  end
...

Reboot
1. ext/export/bin/exportd
2. bin/oddbd
Run Updater.new(self).export_generics_xls from bin/admin

Result

masa@masa ~/ywesee/oddb.org $ cat ~/work/test.dat 
mhatakeyama@ywesee.com

Experiment ext/export/src/generics_xls.rb#export_generics

            def export_generics
open("/home/masa/work/test.dat","w"){|o| o.print RECIPIENTS, "\n"}
exit
...

=begin
begin
      require 'testenvironment2'
rescue LoadError
end
=end

Reboot
1. ext/export/bin/exportd
2. bin/oddbd
Run Updater.new(self).export_generics_xls from bin/admin

Result

masa@masa ~/ywesee/oddb.org $ cat ~/work/test.dat 
mhatakeyama@ywesee.comzdavatz@ywesee.com

Notes

Works well
testenvironment.rb and testenvironment2.rb cannot be merged,
since 'require 'testenvironment'' is called in src/util/oddbapp.rb before loading 'generics_xls.rb'
Ruby constant is overwritten with a warning if the definition is duplicated

Commit

Check again with the latest data today

I got an error

EMail ch.ODDB.org Report - Error: Generikaliste - 12/2010

Plugin: ODDB::XlsExportPlugin
Error: NoMethodError
Message: undefined method `[]' for nil:NilClass
Backtrace:
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:187:in `_remarks'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:178:in `remarks'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:170:in `format_row'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:71:in `export_comparable'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:78:in `export_comparables'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:76:in `each'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:76:in `export_comparables'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:132:in `export_generics'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:132:in `each'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/generics_xls.rb:132:in `export_generics'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:98:in `export_generics_xls'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:255:in `call'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:255:in `safe_export'
(druby://localhost:10005) /usr/lib64/ruby/1.8/tempfile.rb:172:in `open'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:254:in `safe_export'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:96:in `export_generics_xls'
/home/masa/ywesee/oddb.org/src/plugin/xls_export.rb:23:in `export_generics'
/home/masa/ywesee/oddb.org/src/util/updater.rb:101:in `export_generics_xls'
/home/masa/ywesee/oddb.org/src/util/updater.rb:458:in `call'
/home/masa/ywesee/oddb.org/src/util/updater.rb:458:in `wrap_update'
/home/masa/ywesee/oddb.org/src/util/updater.rb:99:in `export_generics_xls'
/home/masa/ywesee/oddb.org/src/util/updater.rb:272:in `update_bsv_followers'
/home/masa/ywesee/oddb.org/src/util/updater.rb:216:in `run'
jobs/import_daily:13
/home/masa/ywesee/oddb.org/src/util/job.rb:17:in `call'
/home/masa/ywesee/oddb.org/src/util/job.rb:17:in `run'
jobs/import_daily:12

Note

This is a different error from before

Experiment

Try run 'export_generics' with 'update_bsv'

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101221-update-ch_oddb_org-generics_xls

Confirm the last night run of export_generics

Implement a function of reporting with skipping nil data

testenvironment.rb

Check again with the latest data today

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101221-update-ch_oddb_org-generics_xls

Confirm the last night run of export_generics

Implement a function of reporting with skipping nil data

testenvironment.rb

Check again with the latest data today

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler