view · edit · sidebar · attach · print · history

20101209-debug-ebook-compendium_ch_oddb_org_firefox_epub

<< Masa.20101210-report-export-fachinfo_yaml | 2010 | Masa.20101208-update-spreadsheet-gem-debug-ebook >>

  1. Tracebak error message
  2. Create a new GnuPG key
  3. Run export_fachinfos on oddb.org
  4. Run 3 epub jobs on epbs
  5. Make a report-mail process suspend
  6. oddb_kindle failed
  7. Screen Command
  8. Make a report-mail process
  9. Make a report-mail process in ch.oddb.org
Goal
  • Debug Ebook / 70%
Milestones
  1. Traceback error message 9:30
  2. Run export_fachinfos on oddb.org
  3. Run 3 epub jobs on ebps 11:30
  4. Understand admin command to delete a Fachinfo on oddb.org
  5. Debug oddb_kindle
  6. Make a report-mail process
Summary
Commits
Attach
ToDo Tomorrow
  • run_ch.oddb_ebooks again by mhatakeyama
  • locally check export_fachinfos
  • Error report process in oddb.org
Keep in Mind
  1. Test case spreadsheet outline
  2. jeweler
  3. YARD + rspec
  4. Author info in code
  5. VirtualBox kernel module
  6. todos
  7. BAG: Files published obviously only on the first ch.ODDB.org Report - Error: Generikaliste - 12/2010 log passwd
  8. Testcase of 'Object Stream' parser (for rpdf2txt/lib/rpdf2txt/parser.rb#rebuild_object_catalogue method) 20101126-update-rpdf2txt
  9. Testcases of lib/oddb/html/state/global.rb#grant_download, lib/oddb/html/view/download.rb#to_html
  10. Debug testcases in test/export/test_server.rb de.oddb.org
  11. A bug import_gkv Tue Nov 16 02:00:10 2010: de.oddb.org Zubef (PDF)
  12. Compression (refer to lib/oddb/export/server.rb), Test cases (grant_download, Logging, Reporting)
  13. Log Error: on production server, de.oddb.org/log/import_dimdi, import_pharmnet
  14. On Ice
  15. emerge --sync

Tracebak error message

Error message 

NoMethodError: undefined method `metadata' for nil:NilClass
/home/masa/work/ebps/lib/ebps/conversion/fachinfo_yaml.rb:8:in `block in <module:EBPS>'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `call'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `transfer'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `node_import'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `load_documents'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `each_document'
/home/masa/work/ebps/lib/ebps/conversion/fachinfo_yaml.rb:83:in `import'
oddb_firefox:58:in `<main>'

Experiment

ebps/lib/ebps/conversion/fachinfo_yaml.rb 

require 'ebps/text/document'
require 'yaml'

module EBPS
  YAML.add_domain_type 'oddb.org,2003', 'ODDB::Fachinfo' do |type, val|
    if descr = val.delete('descriptions')
      doc = descr[EBPS.config.language]
print "doc=", doc, "\n"
      doc.metadata.update val
      doc
    end
  end
...

Result 

...
doc=Pflanzliches Heilmittel bei ...
doc=

Note

  • Too much information

ebps/lib/ebps/conversion/fachinfo_yaml.rb 

require 'ebps/text/document'
require 'yaml'

module EBPS
  YAML.add_domain_type 'oddb.org,2003', 'ODDB::Fachinfo' do |type, val|
    if descr = val.delete('descriptions')
      doc = descr[EBPS.config.language]
print "doc.class=", doc.class, "\n"
      doc.metadata.update val
      doc
    end
  end
...

Result 

...
doc.class=EBPS::Text::Document
doc.class=EBPS::Text::Document
doc.class=EBPS::Text::Document
doc.class=EBPS::Text::Document
doc.class=EBPS::Text::Document
doc.class=NilClass

Hypothesis

  • Probably because of some data missing in fachinfos.yaml

Experiment

ebps/lib/ebps/conversion/fachinfo_yaml.rb 

require 'ebps/text/document'
require 'yaml'

module EBPS
  YAML.add_domain_type 'oddb.org,2003', 'ODDB::Fachinfo' do |type, val|
    if descr = val.delete('descriptions')
      doc = descr[EBPS.config.language]
print "doc.class=", doc.class, "\n"
print "EBPS.config.language=", EBPS.config.language, "\n"
if doc == nil
  p descr
end

Run 

masa@masa ~/work/ebps $ ruby1.9 -I lib oddb_firefox config="config.yml" > test.dat

Result 

EBPS.config.language=de
{"fr"=>OEMéd
Composition
...}

Note

  • There is no value corresponding to 'de' key in the data

Check fachinfos.yaml 

...
--- !oddb.org,2003/ODDB::Fachinfo 
oid: 25419715
descriptions: !oddb.org,2003/ODDB::SimpleLanguage::Descriptions
  fr: !oddb.org,2003/ODDB::FachinfoDocument2001
...

Note

  • In this entry, there is no 'de:' data

Experiment ebps/lib/ebps/conversion/fachinfo_yaml.rb 

require 'ebps/text/document'
require 'yaml'

$masa=0
module EBPS
  YAML.add_domain_type 'oddb.org,2003', 'ODDB::Fachinfo' do |type, val|
    if descr = val.delete('descriptions')
      if(doc = descr[EBPS.config.language])
        doc.metadata.update val
        doc
      else
        doc = descr['fr']
        $masa+=1
        p $masa
        doc.metadata.update val
        doc
      end
    end
  end

Run 

masa@masa ~/work/ebps $ ruby1.9 -I lib oddb_firefox config="config.yml"
1
-> "220 mx.google.com ESMTP b47sm705983wer.14\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-mx.google.com at your service, \r\n"
-> "250-SIZE 35651584\r\n"
...

Result 

Input file for conversion: /home/masa/work/fachinfos.ch.oddb.yaml

The Ebook sample.epub was generated and stored in /home/masa/work

Solution

  1. Search the name of drug on the web, and check the number
  2. Admin command on production server 
reg = @system.registration('32747'); delete reg.fachinfo.pointer; update reg.pointer, :fachinfo => nil

es gibt zwei Möglichkeiten.

1) Wenn eine Andere Registration die gleiche Fachinfo weiterhin anzeigen
soll, dann dürfen wir die FI selber nicht löschen, sondern nur die
Verbindung zwischen Registration und FI:

ch.oddb> update @system.registration('32718').pointer, :fachinfo => nil

2) Wenn wir die Fachinfo nicht mehr brauchen:

ch.oddb> reg = @system.registration('32718'); delete
reg.fachinfo.pointer; update reg.pointer, :fachinfo => nil

(das muss alles auf einer Zeile sein, der Mail-client wirds aber
wahrscheinlich umbrechen...
  1. Reload the search result on the web
  2. Click the fach number, 32747
  3. Check 'FI und PI aktualisieren'
  4. Click 'Speichern'
  5. Then data is updated

Next

  1. Run export fachinfo.ch.oddb.org.yaml
  2. Run 3 export epub jobs
    • /var/ebps/bin/run_ch.oddb_ebooks (oddb_firefox, oddb_stanza, oddb_kindle)

Create a new GnuPG key

my public key

Run export_fachinfos on oddb.org

From oddb.org using bin/admin 

ch.oddb> YamlExporter.new(self).export_fachinfos

Run 3 epub jobs on epbs

Run epub jobs on ebps server with my account 

/var/ebps/bin/run_ch.oddb_ebooks

Result

  • Fails to access to the oddb.org server through ssh

Make ssh key on epbs server 

$ ssh-keygen -t rsa

And

  • register it into authorized_keys of the account on oddb.org server

Run again 

/var/ebps/bin/run_ch.oddb_ebooks

Result 

Input file for conversion: /var/ebps/data/yaml/fachinfos.ch.oddb.yaml

The Ebook compendium_ch.oddb.org.firefox.epub was generated and stored in /var/ebps/data/books

Postprocess: the following commands were executed
 - scp compendium_ch.oddb.org.firefox.epub ywesee@thp:/var/www/oddb.org/data/downloads/compendium_ch.oddb.org.firefox.epub in directory /var/ebps/data/books/

Make a report-mail process

Email yaml-Export Error Mail Notificatio in Case of missing Fachinfo file

  • We should receive and Error-Mail from ch.oddb.org if the yaml-Exporter finds one FI in only one language.
    • Subject: Fachinfos DE or FR is missing for "12345", "Name"
    • Body: The following Fachinfos misses the "DE" or "FR" file. The number of the Fachinfo is "12345". The name of the Fachinfo is "Name".

BraSt

  • I should check first the current error|success mail process on ebps
  • Then add an additional message on it

grep search 

masa@masa ~/work/ebps $ grep -ri mail lib
lib/ebps/config.rb:    'smtp_server'       => 'mail.ywesee.com',
lib/ebps/util/mail.rb:require 'rmail'
lib/ebps/util/mail.rb:module Mail
lib/ebps/util/mail.rb:  def Mail.notify log
lib/ebps/util/mail.rb:    sendmail message, to
lib/ebps/util/mail.rb:  def Mail.sendmail(message, to, cc=[])
lib/ebps/util/mail.rb:      smtp.sendmail(message.to_s, config.smtp_user, [to, cc].flatten.compact)
lib/ebps/util/mail.rb:  def Mail.setup
lib/ebps/util/mail.rb:    message = RMail::Message.new

Note

  • mail sending process must be in mail.rb

suspend

oddb_kindle failed

The success mail does not come from oddb_kindle job

Log 

Info(prcgen): Added metadata dc:Title        "Compendium der Schweizer Arzneimittel von ch.oddb.org fuer Kindle"
Info(prcgen): Added metadata dc:Date         "2010-12-09"
Info(prcgen): Added metadata dc:Creator      "ch.oddb.org"
Info(prcgen): Parsing files  0004151
Info(prcgen): Resolving hyperlinks
Info(prcgen): Building table of content     URL: /tmp/d20101209-23087-mm6b6x/toc.ncx
Warning(index build): string too long (max=255). Truncated.
Warning(index build): string too long (max=255). Truncated.
Info(prcgen): Computing UNICODE ranges used in the book
Info(prcgen): Found UNICODE range: Basic Latin [20..7E]
Info(prcgen): Found UNICODE range: Geometric Shapes [25A0..25FF]
Info(prcgen): Found UNICODE range: Latin-1 Supplement [A0..FF]
Info(prcgen): Found UNICODE range: Letter-like Symbols [2100..214F]
Info(prcgen): Found UNICODE range: General Punctuation - Windows 1252 [2018..201A]
Info(prcgen): Found UNICODE range: Mathematical Operators [2200..22FF]
Info(prcgen): Found UNICODE range: General Punctuation - other than Windows 1252 [2000..2012]
Info(prcgen): Found UNICODE range: Basic Greek [370..3FF]
Info(prcgen): Found UNICODE range: Arrows [2190..21FF]
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000001
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000002
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000004
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000008
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000016
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000032
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000064
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000128
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000256
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0000512
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0001024
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0002048
Info(prcgen/compress): Compiling source text for compression (4096 passes max). Pass  0004095
Info(prcgen/compress): Compression pass  0000001
Write failed: Broken pipe
masa@masa ~/.ssh $ cd

Run it again on ebps server 

/var/ebps/bin $ ./oddb_kindle

Result

Screen Command

Merits

  • Even if 'nohup' is not added, a process keeps running on background after logging out
  • Remote console window can be shared with the other user
  • If there is no X windows system, we can use multi-windows-like screen

start 

 screen 
 screen -t "name"

detach (process keeps to run) 

 c+a, c+d

screen list 

 screen -ls

attach 

 screen -r

attach (shared with the other user) 

 screen -x

If there are some detached screens 

 screen -r|-x [screen ID]

Make a report-mail process

Experiment

lib/ebps/conversion/fachinfo_yaml.rb 

module EBPS
  YAML.add_domain_type 'oddb.org,2003', 'ODDB::Fachinfo' do |type, val|
    if descr = val.delete('descriptions')
      doc = descr[EBPS.config.language]
      begin
      doc.metadata.update val
      rescue NoMethodError => err
        comment = "Probably '#{EBPS.config.language}' data is missing in the following data:\n"
        err.message << "\n\n" << comment << val.to_a.to_s << "\n"
        raise err
      end
      doc
    end
  end

Result 

NoMethodError: undefined method `metadata' for nil:NilClass

Probably 'de' data is missing in the following data:
[["oid", 25419715], ["article_codes", [{:article_pcode=>"1738587", :article_ean13=>"7680327470439"}, {:article_pcode=>"1668898", :article_ean13=>"7680327470354"}, {:article_pcode=>"1668556", :article_ean13=>"7680327470279"}]]]

/home/masa/ywesee/ebps/lib/ebps/conversion/fachinfo_yaml.rb:9:in `block in <module:EBPS>'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `call'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `transfer'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `node_import'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `load_documents'
/usr/lib64/ruby/1.9.1/syck.rb:220:in `each_document'
/home/masa/ywesee/ebps/lib/ebps/conversion/fachinfo_yaml.rb:89:in `import'
oddb_firefox:58:in `<main>'

Commit

Git pull on ebps server 

/usr/local/share/src/ebps $ git pull
remote: Counting objects: 46, done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 38 (delta 19), reused 0 (delta 0)
Unpacking objects: 100% (38/38), done.
From 192.168.0.36:/home/ywesee/git/ebps
   85f01b9..81cabce  master     -> origin/master
Updating 85f01b9..81cabce
Fast-forward
 README.txt                                  |  130 +++-
 example/config.yml                          |   12 +
 example/converter_for_firefox.rb            |   59 ++
 example/data.yml                            |   60 ++
 example/example.sh                          |    3 +
 example/sample.epub                         |  Bin 0 -> 9216 bytes
 lib/ebps/config.rb                          |    1 +
 lib/ebps/conversion/fachinfo_yaml.rb        |    6 +
 lib/ebps/conversion/import_module_sample.rb |   86 ++
 lib/ebps/conversion/oebps.rb                |   12 +-
 lib/ebps/conversion/patinfo_yaml.rb         |  107 +++
 setup.rb                                    | 1345 +++++++++++++++++++++++++++
 12 files changed, 1795 insertions(+), 26 deletions(-)
 create mode 100644 example/config.yml
 create mode 100755 example/converter_for_firefox.rb
 create mode 100644 example/data.yml
 create mode 100755 example/example.sh
 create mode 100644 example/sample.epub
 create mode 100644 lib/ebps/conversion/import_module_sample.rb
 create mode 100644 lib/ebps/conversion/patinfo_yaml.rb
 create mode 100644 setup.rb

Make a report-mail process in ch.oddb.org

Check the current mail process

grep search 

masa@masa ~/ywesee/oddb.org $ grep -r export_fachinfo src
src/plugin/oddbdat_export.rb:   def export_fachinfos
src/plugin/yaml.rb:             def export_fachinfos(name='fachinfo.yaml')
src/util/exporter.rb:        export_fachinfo_pdf
src/util/exporter.rb:    def export_fachinfo_pdf(langs = [:de, :fr])
src/util/exporter.rb:          exporter.export_fachinfos
src/util/exporter.rb:          exporter.export_fachinfos

Notes

  • There is 2 fach_infos methods defined

src/plugin/yaml.rb 

module ODDB
    class YamlExporter < Plugin
        EXPORT_SERVER = DRbObject.new(nil, EXPORT_URI)
        EXPORT_DIR = File.join(ARCHIVE_PATH, 'downloads')
...
        def export_array(name, array, opts={})
            ids = array.collect { |item| item.odba_id }
            EXPORT_SERVER.export_yaml(ids, EXPORT_DIR, name, opts)
        end
...
        def export_fachinfos(name='fachinfo.yaml')
            export_array(name, @app.fachinfos.values)
        end

Notes

  • YamlExporter::export_fachinfos calls export_array, and it calls EXPORT_SERVER.export_yaml

src/util/exporter.rb#export_yaml 

        def export_yaml
            exporter = YamlExporter.new(@app)
...
            run_on_weekday(2) {
        safe_export 'fachinfo.yaml' do
          exporter.export_fachinfos
        end

Notes

  • This method calles YamlExporter.export_fachinfos method

Experiment

src/util/exporter.rb 

    def run
      ## restart the export server
      EXPORT_SERVER.clear
      sleep(30)
      #
=begin
      mail_patinfo_invoices
      mail_fachinfo_log
      run_on_monthday(1) {
        mail_download_invoices
      }
      run_on_monthday(15) {
        mail_download_invoices
      }
      run_on_weekday(0) {
        mail_download_stats
        mail_feedback_stats
        #mail_notification_stats
      }
      export_sl_pcodes
=end
      export_yaml
=begin
      export_oddbdat
      export_csv
      export_doc_csv
      export_index_therapeuticus_csv
      export_price_history_csv
      run_on_monthday(1) {
        export_fachinfo_pdf
      }
=end
      nil
    end

Run jobs/export_daily

Result

  • ch.ODDB.org Report - Error Export: oddb.yaml - 12/2010 
Error: NoMethodError
Message: undefined method `>' for nil:NilClass
Backtrace:
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:147:in `is_binary_data?'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:166:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:165:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:18:in `node_export'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:18:in `add'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:18:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:17:in `each'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:17:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:16:in `map'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:16:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:15:in `to_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/oddb_yaml.rb:14:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `node_export'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `add'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:41:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `each'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:40:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `map'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:39:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `call'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:391:in `quick_emit'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml/rubytypes.rb:38:in `to_yaml'
(druby://localhost:10005) /usr/lib64/ruby/1.8/yaml.rb:117:in `dump'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:235:in `export_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:234:in `each'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:234:in `export_yaml'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:256:in `call'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:256:in `safe_export'
(druby://localhost:10005) /usr/lib64/ruby/1.8/tempfile.rb:172:in `open'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:255:in `safe_export'
(druby://localhost:10005) /home/masa/ywesee/oddb.org/ext/export/src/odba_exporter.rb:233:in `export_yaml'
/home/masa/ywesee/oddb.org/src/plugin/yaml.rb:35:in `export_obj'
/home/masa/ywesee/oddb.org/src/plugin/yaml.rb:12:in `export'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:189:in `export_yaml'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:292:in `call'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:292:in `safe_export'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:188:in `export_yaml'
/home/masa/ywesee/oddb.org/src/util/exporter.rb:52:in `run'
jobs/export_daily:13
/home/masa/ywesee/oddb.org/src/util/job.rb:17:in `call'
/home/masa/ywesee/oddb.org/src/util/job.rb:17:in `run'
jobs/export_daily:12

Contine tomorrow

view · edit · sidebar · attach · print · history
Page last modified on July 13, 2011, at 12:06 PM