view · edit · sidebar · attach · print · history

20101105-create-patinfo-ebook

<< Masa.20101108-sample-ebook-setup-eshop | 2010 | Masa.20101104-update-decorators-for-ebooks >>

Log update_bsv (oddb.org)
Update ebps README
Check the patinfo script from de.oddb.org
Create patinfo script from ch.oddb.org failed
Trace the error message for patinfo.yaml (oddb.org)
Check differences between patinfos.ch.oddb.org and patinfos.de.oddb.org
Update fachinfo_yaml.rb to be able to convert patinfo.yaml

Goal

Create patinfo Ebook from ch.oddb.org / 70%

Milestones

~~Log update_bsv~~ 9:15
~~Update README ebps~~ 10:00
~~Check patinfo script of Ebook from de.oddb.org~~ 14:30
~~Update fachinfo_yaml.rb~~ 16:15
(Update README ebps again, about var, etc, decorator )

Summary

http://masay.ch/2010/11/05/create-patinfo-ebook/

Commits

Added some more logging code for update_bsv (oddb.org)
Updated and arranged README
Added patinfo_yaml.rb for the patinfo convertion of ch.oddb.org (ebps)

ToDo Tomorrow

Update README
Set up E-shop

Keep in Mind

emerge --sync
Rake http://www2s.biglobe.ne.jp/~idesaku/sss/tech/rake/
ch.ODDB.org Report - Error Export: oddbdat - 11/2010 (fwd)
Mon Nov 1 04:09:08 2010: ODDB::Export::Xls::ComparisonDeCh
Update price feeds did not run automatically
Replace get_latest_file (update_swissmedic) method by download_file method
If update_bsv goes well on 1st Dec., let's think about deleting debug print code
delete-first in update_bsv
arrange files and back up script oldie
arrange ydpm job list
xls2odat argument files
Search a fundamental cause of export_chde_xls bug
adding a prefix for the pharmacode field in xls2odat
Delete all SL-Entries before importing the new flags.
20100922-trace-limitation-data-flow-in-bsvXml?action=edit
- 20100922 Options for LimitationNiveau
20100908 rpdf2txt GPLv3
Tokyo Tyrant + ODBA
admin tool hack
Organize the information of yus README
Organize the information of de.oddb.org README
block sample http://q.hatena.ne.jp/1217573703 (japanese)
~~Debug ODDB::CsvExportPlugin~~ Done

Attached Files

Log update_bsv (oddb.org)

2010-11-05 03:01:14 CET getin update_bsv
2010-11-05 03:01:19 CET getin BsvXmlPlugin.update
2010-11-05 03:01:19 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2010-11-05 03:01:19 CET save_dir   = /var/www/oddb.org/data/xml
2010-11-05 03:01:19 CET getin download_file
2010-11-05 03:01:23 CET save_file   = /var/www/oddb.org/data/xml/XMLPublications-2010.11.05.zip
2010-11-05 03:01:23 CET latest_file = /var/www/oddb.org/data/xml/XMLPublications-latest.zip
2010-11-05 03:01:23 CET File.exists?(/var/www/oddb.org/data/xml/XMLPublications-latest.zip) = true
2010-11-05 03:01:23 CET FileUtils.compare_file(/tmp/foo.27874.0, /var/www/oddb.org/data/xml/XMLPublications-latest.zip) = false
2010-11-05 03:01:23 CET path = "/var/www/oddb.org/data/xml/XMLPublications-2010.11.05.zip"

2010-11-05 04:15:55 CET return_value_BsvXmlPlugin.update = "/var/www/oddb.org/data/xml/XMLPublications-2010.11.05.zip"
2010-11-05 04:15:55 CET getin log_notify_bsv
2010-11-05 04:15:55 CET date=Mon, 01 Nov 2010
2010-11-05 04:15:55 CET after pointer creating
2010-11-05 04:16:57 CET after log_info(plug)
2010-11-05 04:16:57 CET before @app.update
2010-11-05 04:16:59 CET after @app.update
2010-11-05 04:16:59 CET getin Log.notify (SL-Update)
2010-11-05 04:16:59 CET @files={}
2010-11-05 04:16:59 CET start outgoing process
2010-11-05 04:16:59 CET before send_mail(outgoing)

2010-11-05 04:17:10 CET getin Log.notify (SL-Update)
2010-11-05 04:17:10 CET @files={}
2010-11-05 04:17:10 CET start outgoing process
2010-11-05 04:17:10 CET before send_mail(outgoing)
2010-11-05 04:17:10 CET return_value_update_bsv=nil

Notes

It seems that the update_bsv runs correctly but the update_bsv_followers did not run
'before send_mail(outgoing)' indicates that the sending data in the process was almost done
The point where the script stopping is src/util/log.rb#send_mail method:

Commit

Added some more logging code for update_bsv (oddb.org)

Update ebps README

Commit

Updated and arranged README

Check the patinfo script from de.oddb.org

Comparison between de_meddrugs_patinfo_firefox and meddrugs.ch_de_firefox

These are totally same

$ diff de_meddrugs_patinfo_firefox meddrugs.ch_de_firefox
(nothing is outputted)

Comparison between de_meddrugs_patinfo_firefox.yml and meddrugs.ch_de_firefox.yml

Fundamentally, the souce yaml files are different.

$ diff de_meddrugs_patinfo_firefox.yml meddrugs.ch_de_firefox.yml 
5c7
< report_to: 
---
> report_to:
8,12c10,14
< author: med-drugs.de
< source: /var/ebps/data/yaml/patinfos.de.oddb.yaml
< target: /var/ebps/data/books/patinfos_de.meddrugs.firefox.epub
< cover: /var/ebps/data/covers/LOGO_startseite_MEDDRUGS_DEU.jpg
< import_from: de_fachinfo_yaml
---
> author: med-drugs.ch
> source: /var/ebps/data/yaml/fachinfos.ch.oddb.yaml
> target: /var/ebps/data/books/meddrugs_ch_firefox.epub
> cover: /var/ebps/data/covers/startseite_MEDDRUGS_CH.jpg
> import_from: fachinfo_yaml
15c17
< image_prefix: /var/www/de.oddb.org/var/
---
> image_prefix: /var/www/oddb.org/doc/
20,21c22,23
< title: Patienteninformationen der Deutschen Arzneimittel von Just-Medical.de fuer Firefox
< appendix: /var/ebps/data/LOGO_endseite_MEDDRUGS_DEU.jpg
---
> title: Schweizer Medikamenten-Enzyklop�die
> appendix: /var/ebps/data/covers/endseite_MEDDRUGS_CH.jpg
25,26c27,31
---
> decorate:
>  - - /var/ebps/bin/decorators/just_medical_de.rb
>    - JustMedical

But there are only @3@ yaml files on production server

/var/ebps/data/yaml $ ls
fachinfos.ch.oddb.yaml  fachinfos.de.oddb.yaml  patinfos.de.oddb.yaml

Notes

We need patinfos.ch.oddb.yaml (probably)

Question

How are these files produced?

Answer

oddb.org/src/plugin/yaml.rb (ODDB::YamlExporter)
This class is called from src/util/exporter.rb#export_yaml which runs everyday

Create patinfo script from ch.oddb.org

Test a script by using patinfo.yaml on production server

meddrugs_patinfo_firefox.yml

---
smtp_server: smtp.gmail.com
smtp_user: xxx@ywesee.com
smtp_pass: 'xxxx'
report_to:
 - mhatakeyama@ywesee.com
author: med-drugs.de
#source: /var/ebps/data/yaml/patinfos.de.oddb.yaml
source: /home/masa/ywesee/ebps/patinfos.ch.oddb.yaml
#target: /var/ebps/data/books/patinfos_de.meddrugs.firefox.epub
target: /home/masa/ywesee/ebps/patinfos_de.meddrugs.firefox.epub
cover: /var/ebps/data/covers/LOGO_startseite_MEDDRUGS_DEU.jpg
import_from: de_fachinfo_yaml
export_to: epub
html_index: true
image_prefix: /var/www/de.oddb.org/var/
image_suffix: .png
language: de
max_depth: 1
stylesheet: /var/ebps/data/css/oddb.css
title: Patienteninformationen der Deutschen Arzneimittel von Just-Medical.de fuer Firefox
appendix: /var/ebps/data/LOGO_endseite_MEDDRUGS_DEU.jpg
xml_indent: 0

Result

Email Ebook patinfos_de.meddrugs.firefox.epub could not be completed (05.11.2010 11:46:59)

Input file for conversion: /home/masa/ywesee/ebps/patinfos.ch.oddb.yaml

NoMethodError: undefined method `title' for #<Syck::DomainType:0x00000001bc8eb8>
/home/masa/ywesee/ebps/lib/ebps/conversion/oebps.rb:436:in `block (2 levels) in export'
/home/masa/ywesee/ebps/lib/ebps/conversion/oebps.rb:435:in `each'
/home/masa/ywesee/ebps/lib/ebps/conversion/oebps.rb:435:in `sort_by'
/home/masa/ywesee/ebps/lib/ebps/conversion/oebps.rb:435:in `block in export'
/home/masa/ywesee/ebps/lib/ebps/conversion/oebps.rb:484:in `call'
/home/masa/ywesee/ebps/lib/ebps/conversion/oebps.rb:484:in `with_tmpdir'
/home/masa/ywesee/ebps/lib/ebps/conversion/oebps.rb:431:in `export'
/home/masa/ywesee/ebps/lib/ebps/conversion/epub.rb:32:in `export'
de_meddrugs_patinfo_firefox:72:in `<main>'

Check patinfos.ch.oddb.yaml

There is no 'title' data in patinfos.ch.oddb.yaml
Next: Check the difference between structures of patinfos.ch.oddb.yaml and patinfos.de.oddb.yaml

Trace the error message for patinfo.yaml (oddb.org)

Setting

Download patinfo.yaml from ch.oddb.org
Modify meddrugs.ch_de_firefox.yml

smtp_server: smtp.gmail.com
smtp_user: xxx@ywesee.com
smtp_pass: 'xxx'
report_to:
 - mhatakeyama@ywesee.com
author: med-drugs.ch
#source: /var/ebps/data/yaml/fachinfos.ch.oddb.yaml
source: /home/masa/work/patinfo.yaml
#target: /var/ebps/data/books/meddrugs_ch_firefox.epub
target: /home/masa/work/meddrugs_ch_firefox.epub
cover: /var/ebps/data/covers/startseite_MEDDRUGS_CH.jpg
import_from: fachinfo_yaml
export_to: epub
html_index: true
image_prefix: /var/www/oddb.org/doc/
image_suffix: .png
language: de
max_depth: 1
stylesheet: /var/ebps/data/css/oddb.css
title: Schweizer Medikamenten-Enzyklop�die
appendix: /var/ebps/data/covers/endseite_MEDDRUGS_CH.jpg
xml_indent: 0

Run meddrugs.ch_de_firefox

$ ruby1.9 -I ebps/lib/ meddrugs.ch_de_firefox config="meddrugs.ch_de_firefox.yml"

Result (Error Mail) Ebook meddrugs_ch_firefox.epub could not be completed (05.11.2010 13:55:54)

Input file for conversion: /home/masa/work/patinfo.yaml

NoMethodError: undefined method `title' for #<Syck::DomainType:0x0000000140e100>
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:436:in `block (2 levels) in export'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `each'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `sort_by'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:435:in `block in export'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:484:in `call'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:484:in `with_tmpdir'
/home/masa/work/ebps/lib/ebps/conversion/oebps.rb:431:in `export'
/home/masa/work/ebps/lib/ebps/conversion/epub.rb:32:in `export'
meddrugs.ch_de_firefox:72:in `<main>'

Trace

1. lib/ebps/conversion/oebps.rb:431

      def self.export docs, target, override_tmpdir=nil
        ids = []
        with_tmpdir do |tmpdir|                         #<= here
          tmpdir = override_tmpdir || tmpdir
          # HTML
          if EBPS.config.sort
            docs = docs.sort_by do |doc|
              doc.title.sortable
            end
          end
...

2. lib/ebps/conversion/oebps.rb:484

      def self.with_tmpdir &block
        tmpdir = Dir.mktmpdir
        block.call tmpdir        #<= here
      ensure
        FileUtils.rm_r tmpdir unless EBPS.config.keep_tmpfiles
      end

3. lib/ebps/conversion/oebps.rb:435

      def self.export docs, target, override_tmpdir=nil
        ids = []
        with_tmpdir do |tmpdir|
          tmpdir = override_tmpdir || tmpdir
          # HTML
          if EBPS.config.sort
            docs = docs.sort_by do |doc|       #<= here and
              doc.title.sortable               #<= here
            end
          end

Experiment

lib/ebps/conversion/oebps.rb:435

      def self.export docs, target, override_tmpdir=nil
        ids = []
        with_tmpdir do |tmpdir|
          tmpdir = override_tmpdir || tmpdir
          # HTML
          if EBPS.config.sort
            docs = docs.sort_by do |doc|
p doc.class
p doc.title
              doc.title.sortable
            end
          end

Run meddrugs.ch_de_firefox with fachinfo.yaml

EBPS::Text::Document
"Nalbuphin OrPha"

Run meddrugs.ch_de_firefox with patinfo.yaml

Syck::DomainType
(Error)

Notes

The classes are different, even though the setting yaml file is same
I guess the class is probably automatically selected depending on yaml data file structure

Check differences between patinfos.ch.oddb.org and patinfos.de.oddb.org

fachinfos.ch.oddb.yaml (source of meddrugs.ch_de_firefox)
patinfos.de.oddb.yaml (source of de_meddrugs_patinfo_firefox)
patinfo.yaml (downloaded from ch.oddb.org)

Reference

YAML RDoc http://ruby-doc.org/stdlib/libdoc/yaml/rdoc/index.html
YAML Reference (Japanese) http://www.ruby-lang.org/ja/man/html/YAML.html

1. fachinfos.ch.oddb.yaml and 2. patinfos.de.oddb.yaml

fachinfos.ch.oddb.yaml

--- !oddb.org,2003/ODDB::Fachinfo 
oid: 25050766
descriptions: !oddb.org,2003/ODDB::SimpleLanguage::Descriptions
  de: !oddb.org,2003/ODDB::FachinfoDocument2001
    name: Nalbuphin OrPha
    amzv: !oddb.org,2003/ODDB::Text::Chapter
      heading: AMZV
      sections: []

    composition: !oddb.org,2003/ODDB::Text::Chapter
      heading: Zusammensetzung
      sections: 
...

patinfos.de.oddb.yaml

--- !de.oddb.org,2007/ODDB::Util::M10lDocument 
oid: 289049
previous_sources:
  :de: []

canonical:
  :de: !de.oddb.org,2007/ODDB::Text::Document
    oid: 289051
    chapters:
    - !de.oddb.org,2007/ODDB::Text::Chapter
      name: composition
      paragraphs:
      - !de.oddb.org,2007/ODDB::Text::Paragraph
        text: "Ate Lich\xC2\xAE 25 mg"
        formats:
        - !de.oddb.org,2007/ODDB::Text::Format
          end: -1
          start: 0
          values:
          - b
...

Notes

Totally different

1. fachinfos.ch.oddb.yaml and 3. patinfo.yaml

fachinfos.ch.oddb.yaml

--- !oddb.org,2003/ODDB::Fachinfo 
oid: 25050766
descriptions: !oddb.org,2003/ODDB::SimpleLanguage::Descriptions
  de: !oddb.org,2003/ODDB::FachinfoDocument2001
    name: Nalbuphin OrPha
    amzv: !oddb.org,2003/ODDB::Text::Chapter
      heading: AMZV
      sections: []

    composition: !oddb.org,2003/ODDB::Text::Chapter
      heading: Zusammensetzung
      sections: 
...

patinfo.yaml

--- !oddb.org,2003/ODDB::Patinfo 
oid: 516
descriptions: !oddb.org,2003/ODDB::SimpleLanguage::Descriptions
  fr: !oddb.org,2003/ODDB::PatinfoDocument
    name: "Isoket\xC2\xAE Spray"
    company: !oddb.org,2003/ODDB::Text::Chapter
      heading: SCHWARZ
      sections: []

    galenic_form: !oddb.org,2003/ODDB::Text::Chapter
      heading: ""
      sections: []
...

Notes

Looks similar but different
Expecially, fachinfo.yaml starts with '2003/ODDB::FachinfoDocument2001'
But patinfo.yaml starts with '2003/ODDB::PatinfoDocument'
It seems that these file structure is converted in ebps/lib/ebps/conversion/*.rb
(fachinfo.yaml (ch.oddb.org) is converted by ebps/lib/ebps/conversion/fachinfo_yaml.rb)

Update fachinfo_yaml.rb to be able to convert patinfo.yaml

Preparing

patinfos.ch.oddb.yaml (downloaded from ch.oddb.org)
meddrugs_patinfo.ch_de_firefox (totally same as meddrugs.ch_de_firefox)
Attach:patinfo_yaml.rb (saved in lib/ebps/conversion)
meddrugs_patinfo.ch_de_firefox.yml (as follows)

---
smtp_server: smtp.gmail.com
smtp_user: xxx@ywesee.com
smtp_pass: 'xxx'
report_to:
 - mhatakeyama@ywesee.com
author: med-drugs.ch
source: /home/masa/work/patinfos.ch.oddb.yaml
#target: /var/ebps/data/books/meddrugs_ch_firefox.epub
target: /home/masa/work/meddrugs_patinfo_ch_de_firefox.epub
cover: /var/ebps/data/covers/startseite_MEDDRUGS_CH.jpg
import_from: patinfo_yaml
export_to: epub
html_index: true
image_prefix: /var/www/oddb.org/doc/
image_suffix: .png
language: de
max_depth: 1
stylesheet: /var/ebps/data/css/oddb.css
title: Schweizer Medikamenten-Enzyklop�die
appendix: /var/ebps/data/covers/endseite_MEDDRUGS_CH.jpg
xml_indent: 0

Run

masa@masa ~/work $ ruby1.9 -I ebps/lib meddrugs_patinfo.ch_de_firefox config="meddrugs_patinfo.ch_de_firefox.yml"

Result

Success
Uploaded on just-medical.com tentatively named as 'meddrugs_patinfo_ch_de_firefox.epub'

Commit

Added patinfo_yaml.rb for the patinfo convertion of ch.oddb.org

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101105-create-patinfo-ebook

Log update_bsv (oddb.org)

Update ebps README

Check the patinfo script from de.oddb.org

Create patinfo script from ch.oddb.org

Trace the error message for patinfo.yaml (oddb.org)

Trace

Check differences between patinfos.ch.oddb.org and patinfos.de.oddb.org

1. fachinfos.ch.oddb.yaml and 2. patinfos.de.oddb.yaml

1. fachinfos.ch.oddb.yaml and 3. patinfo.yaml

Update fachinfo_yaml.rb to be able to convert patinfo.yaml

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101105-create-patinfo-ebook

Log update_bsv (oddb.org)

Update ebps README

Check the patinfo script from de.oddb.org

Create patinfo script from ch.oddb.org

Trace the error message for patinfo.yaml (oddb.org)

Trace

Check differences between patinfos.ch.oddb.org and patinfos.de.oddb.org

1. fachinfos.ch.oddb.yaml and 2. patinfos.de.oddb.yaml

1. fachinfos.ch.oddb.yaml and 3. patinfo.yaml

Update fachinfo_yaml.rb to be able to convert patinfo.yaml

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler