view · edit · sidebar · attach · print · history

20101210-report-export-fachinfo_yaml

<< Masa.20101213-update-export_fachinfo_yaml | 2010 | Masa.20101209-debug-ebook-compendium_ch_oddb_org_firefox_epub >>

Check Ebooks result yesterday
Check export_fachinfos process locally
Make a report process

Goal

Report process fachinfo.yaml / 80%

Milestones

~~Check ebooks result~~ 8:00
~~Check export fachinfo.yaml locally~~ 11:40

Update error report process in oddb.org

Summary

http://masay.ch/2010/12/10/report-export-fachinfo_yaml/

Commits
ToDo Tomorrow
Keep in Mind

Test case spreadsheet outline
jeweler
YARD + rspec
Author info in code
VirtualBox kernel module
todos
BAG: Files published obviously only on the first ch.ODDB.org Report - Error: Generikaliste - 12/2010 log passwd
Testcase of 'Object Stream' parser (for rpdf2txt/lib/rpdf2txt/parser.rb#rebuild_object_catalogue method) 20101126-update-rpdf2txt
Testcases of lib/oddb/html/state/global.rb#grant_download, lib/oddb/html/view/download.rb#to_html
Debug testcases in test/export/test_server.rb de.oddb.org
A bug import_gkv Tue Nov 16 02:00:10 2010: de.oddb.org Zubef (PDF)
Compression (refer to lib/oddb/export/server.rb), Test cases (grant_download, Logging, Reporting)
Log Error: on production server, de.oddb.org/log/import_dimdi, import_pharmnet
On Ice
emerge --sync

Check Ebooks result yesterday

Email Ebook compendium_ch.oddb.org.kindle.mobi finished at 09.12.2010 20:48:53

Input file for conversion: /var/ebps/data/yaml/fachinfos.ch.oddb.yaml

The Ebook compendium_ch.oddb.org.kindle.mobi was generated and stored in /var/ebps/data/books

Postprocess: the following commands were executed
 - scp compendium_ch.oddb.org.kindle.mobi xxx@zzz:/var/www/oddb.org/data/downloads/compendium_ch.oddb.org.kindle.mobi in directory /var/ebps/data/books/
 - scp compendium_ch.oddb.org.kindle.mobi xxx@zzz:/var/www/oddb.org/data/downloads/compendium_ch.oddb.org.htc.prc in directory /var/ebps/data/books/

/var/ebps/data/books $ ls -al
total 480920
-rwxrwxr-x 1 hwyss   ywesee  31129432 Dec  9 11:23 compendium_ch.oddb.org.firefox.epub
-rwxrwxr-x 1 hwyss   ywesee  15706692 Dec  9 20:48 compendium_ch.oddb.org.kindle.mobi
-rwxrwxr-x 1 hwyss   ywesee  30705118 Dec  9 11:29 compendium_ch.oddb.org.stanza.epub

Note

Looks good

Run oddb_firefox on ebps server again

/var/ebps/bin $ ./oddb_firefox

Result

Email Ebook compendium_ch.oddb.org.firefox.epub finished at 10.12.2010 08:23:36

Input file for conversion: /var/ebps/data/yaml/fachinfos.ch.oddb.yaml

The Ebook compendium_ch.oddb.org.firefox.epub was generated and stored in /var/ebps/data/books

Postprocess: the following commands were executed
 - scp compendium_ch.oddb.org.firefox.epub xxx@zzz:/var/www/oddb.org/data/downloads/compendium_ch.oddb.org.firefox.epub in directory /var/ebps/data/books/

mhatakeyama@fastpower /var/ebps/data/books $ ls -al
total 480920
drwxrwxr-x 2 hwyss   ywesee      4096 Jul  9 08:38 .
drwxrwxr-x 7 hwyss   ywesee        64 Jun 11 13:22 ..
-rwxrwxr-x 1 hwyss   ywesee  31129432 Dec 10 08:23 compendium_ch.oddb.org.firefox.epub
-rwxrwxr-x 1 hwyss   ywesee  15706692 Dec  9 20:48 compendium_ch.oddb.org.kindle.mobi
-rwxrwxr-x 1 hwyss   ywesee  30705118 Dec  9 11:29 compendium_ch.oddb.org.stanza.epub

Note

Looks good too

Check export_fachinfos process locally

Experiment

Run oddb.org/bin/oddbd
Run oddb.org/ext/export/bin/exportd

src/util/exporter.rb

    def run
      ## restart the export server
      EXPORT_SERVER.clear
      sleep(30)
      #
=begin
      mail_patinfo_invoices
      mail_fachinfo_log
      run_on_monthday(1) {
        mail_download_invoices
      }
      run_on_monthday(15) {
        mail_download_invoices
      }
      run_on_weekday(0) {
        mail_download_stats
        mail_feedback_stats
        #mail_notification_stats
      }
      export_sl_pcodes
=end
      export_yaml
=begin
      export_oddbdat
      export_csv
      export_doc_csv
      export_index_therapeuticus_csv
      export_price_history_csv
      run_on_monthday(1) {
        export_fachinfo_pdf
      }
=end
      nil
    end

...
        def export_yaml
            exporter = YamlExporter.new(@app)
=begin
      safe_export 'oddb.yaml' do
        exporter.export
      end
      safe_export 'atc.yaml' do
        exporter.export_atc_classes
      end
      safe_export 'interactions.yaml' do
        exporter.export_interactions
      end
      safe_export 'narcotics.yaml' do
        exporter.export_narcotics
      end
      safe_export 'price_history.yaml' do
        exporter.export_prices
      end
            run_on_weekday(2) {
=end
        safe_export 'fachinfo.yaml' do
          exporter.export_fachinfos
        end
=begin
...
=end
      end

Run jobs/export_daily

Result

Email ch.ODDB.org Report - Error Export: fachinfo.yaml - 12/2010

Error: NoMethodError
Message: undefined method `>' for nil:NilClass
Backtrace:
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:247:in `node_export'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:247:in `add'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:247:in `to_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:246:in `each'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:246:in `to_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:245:in `map'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:245:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:244:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:234:in `export_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:233:in `each'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:233:in `export_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:255:in `call'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:255:in `safe_export'
(druby://localhost:10005) /usr/lib64/ruby/1.8/tempfile.rb:172:in `open'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:254:in `safe_export'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:232:in `export_yaml'
/home/masa/ywesee/oddb.org/src/plugin/yaml.rb:16:in `export_array'
/home/masa/ywesee/oddb.org/src/plugin/yaml.rb:25:in `export_fachinfos'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:206:in `export_yaml'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:295:in `call'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:295:in `safe_export'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:205:in `export_yaml'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:52:in `run'
jobs/export_daily:13
/home/masa/ywesee/oddb.org/src/util/job.rb:17:in `call'
/home/masa/ywesee/oddb.org/src/util/job.rb:17:in `run'
jobs/export_daily:12

Quesion

Why does this stop?

Memo

The calling order of methods is as follows:

Exporter#export_yaml
YamlExporter#export_fachinfos
YamlExporter#export_array
OdbaExporter#export_yaml
Fachinfo#to_yaml

Experiment (simple YAML test)

require 'yaml'

class Test
  def initialize
    @a = [1,2,3]
  end
end

test=Test.new
puts YAML::dump(test)

Result

masa@masa ~/work $ ruby test.rb 
/usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?': undefined method `>' for nil:NilClass (NoMethodError)
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:18:in `node_export'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:18:in `add'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:18:in `to_yaml'
         ... 6 levels...
        from /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
        from /usr/lib64/ruby/1.8/yaml/rubytypes.rb:15:in `to_yaml'
        from /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
        from test.rb:11

Note

This is the same error as export_fachinfo method

Experiment

/usr/lib64/ruby/1.8/yaml/rubytypes.rb

    def is_binary_data?
        ( self.count( "^ -~", "^\r\n" ) / self.size > 0.3 || self.count( "\x00" ) > 0 ) unless empty?
        #( self.count( "^ -~", "^\r\n" ) / self.size > 0.3 || self.index( "\x00" ) > 0 ) unless empty?
    end

Note

This part is the change that I updated on 29 July 2010 for YAML bug of oddb.org
Refer to http://masay.ch/2010/07/29/look-into-yaml/

Result

masa@masa ~/work $ ruby test.rb 
--- !ruby/object:Test 
a: 
- 1
- 2
- 3

Note

It works...

I have got my mistake!!

In Ruby 1.8.6, I need to edit /usr/lib64/ruby/1.8/yaml/rubytypes.rb as follows:
This has been corrected in Ruby 1.8.7 and Ruby 1.9

   def is_binary_data?
        #( self.count( "^ -~", "^\r\n" ) / self.size > 0.3 || self.count( "\x00" ) > 0 ) unless empty?
        #( self.count( "^ -~", "^\r\n" ) / self.size > 0.3 || self.index( "\x00" ) > 0 ) unless empty?
        ( self.count( "^ -~", "^\r\n" ) / self.size > 0.3 || self.index( "\x00" ) ) unless empty?
    end

Note

It should be like that

Try to run oddb.org/jobs/export_daily again

Result

There is no success report email but
Success

masa@masa ~/ywesee/oddb.org $ ls data/downloads/ -al
insgesamt 502420
drwxr-xr-x 2 masa masa        56 10. Dez 11:18 .
drwxr-xr-x 8 masa masa        48 10. Dez 08:09 ..
-rw-r--r-- 1 masa masa 393117616 10. Dez 11:13 fachinfo.yaml
-rw-r--r-- 1 masa masa  60674662 10. Dez 11:18 fachinfo.yaml.gz
-rw-r--r-- 1 masa masa  60674768 10. Dez 11:18 fachinfo.yaml.zip

Make a report process

How to check 'fr' and 'de' entry in YAML file

How about checking in the following method

ext/export/src/odba_exporter.rb

        def OdbaExporter.export_yaml(odba_ids, dir, name, opts={})
      opts.each do |key, val| Thread.current[key] = val end
            safe_export(dir, name) { |fh|
                odba_ids.each { |odba_id|
                    YAML.dump(ODBA.cache.fetch(odba_id, nil), fh)
                    fh.puts
                }
                nil
            }
        end

Note

In this method, ODBA object is fetched and dumped

Experiment (if I can check 'fr' or 'de' entry from fetched object)

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> ODBA.cache.fetch(23706967,nil)
-> #<ODDB::FachinfoDocument2001:0x7fba4eb6e530>
ch.oddb> ODBA.cache.fetch(23706967,nil).methods
-> Array

grep search

masa@masa ~/ywesee/oddb.org $ grep -ri fachinfodocument src
src/model/fachinfo.rb:  class FachinfoDocument
src/model/fachinfo.rb:  class FachinfoDocument2001 < FachinfoDocument
src/state/admin/user.rb:                doc = FachinfoDocument2001.new
src/state/admin/wait_for_fachinfo.rb:           if(document.is_a?(FachinfoDocument))
src/util/oddbapp.rb:    when ODDB::Fachinfo, ODDB::FachinfoDocument
src/view/rss/fachinfo.rb:        next unless(document.is_a?(FachinfoDocument))

ch.oddb> ODBA.cache.fetch(23706967,nil).descriptions['aa'].class
-> NilClass
ch.oddb> ODBA.cache.fetch(23706967,nil).descriptions['fr'].class
-> ODDB::FachinfoDocument2001
ch.oddb> ODBA.cache.fetch(23706967,nil).descriptions['de'].class
-> ODDB::FachinfoDocument2001

Notes

If there is no description corresponding to 'de' language, the 'descriptions['de']' of FachinfoDocument object becomes 'nil'
I can use that property for checking the 'de' or 'fr' entries

Note

If there is not 'de' or 'fr' description on the ODBA cache, the first description is returned without error
See below

        def description(key=nil)
            descriptions[key.to_s] || descriptions.first
        end

Example

ch.oddb> ODBA.cache.fetch(23706967,nil).search_text('de')
-> IndikationenAnwendungsmoeglichkeiten Prophylaktische ...
ch.oddb> ODBA.cache.fetch(23706967,nil).search_text('fr')
-> IndicationsPossibilites d�emploi Traitement prophylactique ...
ch.oddb> ODBA.cache.fetch(23706967,nil).search_text('jp')
-> IndikationenAnwendungsmoeglichkeiten Prophylaktische ...

It is possible to check also here

src/plugin/yaml.rb

        def export_fachinfos(name='fachinfo.yaml')
            export_array(name, @app.fachinfos.values)
        end

@app.fachinfos.class is Hash, and key:odba_id, value:ODDB::FachinfoDocument2001

Question

Can I get the EAN code corresponding to an odba_id?

Experiment

src/plugin/yaml.rb

        def export_fachinfos(name='fachinfo.yaml')
$node = 0
$nofr = 0
open("/home/masa/work/log.dat","a"){|o|
            @app.fachinfos.values.each do |fachinfodocument|
              unless fachinfodocument.descriptions['de']
                o.print "no de decsription odba_id= ", fachinfodocument.odba_id, "\n"
                $node += 1
              end
              unless fachinfodocument.descriptions['fr']
                o.print "no fr description odba_id=: ", fachinfodocument.odba_id, "\n"
                $nofr += 1
              end
            end
            o.print "TOTAL:\n"
            o.print "no de: ", $node, "\n"
            o.print "no fr: ", $nofr, "\n"
}
exit
            export_array(name, @app.fachinfos.values)
        end

Result

no fr description odba_id=: 644479
no fr description odba_id=: 1458238
no fr description odba_id=: 17054174
no fr description odba_id=: 879026
TOTAL:
no de: 0
no fr: 4

Note

I can get the entries of fachinfo without 'fr' description
I do not want to stop the export_fachinfos process because of this error, but
I want the warning report

Experiment

src/plugin/yaml.rb

        def export_fachinfos(name='fachinfo.yaml')
$node = 0
$nofr = 0
message = []
            @app.fachinfos.values.each do |fachinfo|
              unless fachinfo.descriptions['de']
                message << "no de decsription odba_id= " + fachinfo.odba_id.to_s
                message << "name_base= " + fachinfo.name_base.to_s
                $node += 1
              end
              unless fachinfo.descriptions['fr']
                message << "no fr decsription odba_id= " + fachinfo.odba_id.to_s
                message << "name_base= " + fachinfo.name_base.to_s
                $nofr += 1
              end
            end
            message << "TOTAL" << "no de: " + $node.to_s << "no fr: " + $nofr.to_s

if $node+$nofr > 0
  log = Log.new(@@today)
  log.report = "Message: " + message.join("\n")
  log.notify(" Warning Export: #{name}")
end
            export_array(name, @app.fachinfos.values)
        end

Result

Email ch.ODDB.org Report - Warning Export: fachinfo.yaml - 12/2010

Message: no fr decsription odba_id= 644479
name_base= Amicosol 1,2 g (1000/200)
no fr decsription odba_id= 1458238
name_base= Muco-X 200
no fr decsription odba_id= 17054174
name_base= Imucal
no fr decsription odba_id= 879026
name_base= Gen H-B-Vax 5
TOTAL
no de: 0
no fr: 4

Looks not bad
export_fachinfos process still runs even if this email comes

Arrange the message

Experiment

src/plugin/yaml.rb

        def export_fachinfos(name='fachinfo.yaml')
no_descr = {'de' => [], 'fr' => []}
            @app.fachinfos.values.each do |fachinfo|
              swissmedic_registration_numbers = ODBA.cache.fetch(fachinfo.odba_id, nil).iksnrs
              no_descr.keys.each do |language|
                unless fachinfo.descriptions[language]
                 no_descr[language].concat swissmedic_registration_numbers
                end
              end
            end
if no_descr.values.flatten.length > 0
  log = Log.new(@@today)
  message = []
  no_descr.keys.each do |language|
    unless no_descr[language].empty?
      message.concat([
        "There is not '#{language}' description of the Fach Information of the following",
        "Swissmedic Registration Number:",
        no_descr[language].join(", ").to_s
      ])
    end
  end

Result

Email ch.ODDB.org Report - Error Export: fachinfo.yaml - 12/2010

Message:
YamlExporter#export_fachinfs method is still running,
but I found some missing Fachinfo document data.
This may cause an error in export ebooks process of ebps.

There is no 'fr' description of Fachinformation for the following
Swissmedic Registration Numbers:
56811, 58417, 53183, 00610, 00528

Commit

http://scm.ywesee.com/?p=oddb.org/.git;a=commit;h=b0b00eb0655f58d3329f51966d9c15340783d49a

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101210-report-export-fachinfo_yaml

Check Ebooks result yesterday

Check export_fachinfos process locally

Make a report process

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101210-report-export-fachinfo_yaml

Check Ebooks result yesterday

Check export_fachinfos process locally

Make a report process

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler