view · edit · sidebar · attach · print · history

20101103-decorators-for-ebooks

<< Masa.20101104-update-decorators-for-ebooks | 2010 | Masa.20101102-debug-oddb-csv_export >>

Check update_bsv log last night
Check mail sending process and put some logging code
Set up Ebps
Update the decorator of meddrugs.ch_de_firefox

Goal

Decorators for Ebooks / 70%

Milestones

~~Check the log of update_bsv yesterday~~
~~Put logging code more~~ 10:20
~~Ebps: build the same condition~~ 14:15
Update decorator
- I have to read and understand the code

Summary

http://masay.ch/2010/11/03/decorators-for-ebooks/

Commits

Added more logging code for checking update_bsv process (oddb.org)
Added setup.rb for installation (ebps)

ToDo Tomorrow
Keep in Mind

emerge --sync
Debug ODDB::CsvExportPlugin
ch.ODDB.org Report - Error Export: oddbdat - 11/2010 (fwd)
Mon Nov 1 04:09:08 2010: ODDB::Export::Xls::ComparisonDeCh
Update price feeds did not run automatically
Replace get_latest_file (update_swissmedic) method by download_file method
If update_bsv goes well on 1st Dec., let's think about deleting debug print code
delete-first in update_bsv
arrange files and back up script oldie
arrange ydpm job list
xls2odat argument files
Search a fundamental cause of export_chde_xls bug
adding a prefix for the pharmacode field in xls2odat
Delete all SL-Entries before importing the new flags.
20100922-trace-limitation-data-flow-in-bsvXml?action=edit
- 20100922 Options for LimitationNiveau
20100908 rpdf2txt GPLv3
Tokyo Tyrant + ODBA
admin tool hack
Organize the information of yus README
Organize the information of de.oddb.org README
block sample http://q.hatena.ne.jp/1217573703 (japanese)

Attached Files

Check update_bsv log last night

2010-11-03 03:01:14 CET getin update_bsv
2010-11-03 03:01:19 CET getin BsvXmlPlugin.update
2010-11-03 03:01:19 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2010-11-03 03:01:19 CET save_dir   = /var/www/oddb.org/data/xml
2010-11-03 03:01:19 CET getin download_file
2010-11-03 03:01:21 CET save_file   = /var/www/oddb.org/data/xml/XMLPublications-2010.11.03.zip
2010-11-03 03:01:21 CET latest_file = /var/www/oddb.org/data/xml/XMLPublications-latest.zip
2010-11-03 03:01:21 CET File.exists?(/var/www/oddb.org/data/xml/XMLPublications-latest.zip) = false
2010-11-03 03:01:21 CET path = "/var/www/oddb.org/data/xml/XMLPublications-2010.11.03.zip"
2010-11-03 04:11:01 CET return_value_BsvXmlPlugin.update = "/var/www/oddb.org/data/xml/XMLPublications-2010.11.03.zip"
2010-11-03 04:11:01 CET getin log_notify_bsv
2010-11-03 04:12:05 CET getin Log.notify (SL-Update)
2010-11-03 04:12:17 CET getin Log.notify (SL-Update)
2010-11-03 04:12:17 CET return_value_update_bsv=nil

Notes

update_bsv looked running, but email reports did not come
Davatz-san got only the first mail but I did not.

Questions

Why did only the first mail come to only to Davatz-san?
Where is the process stopping?

Export_csv fails on production server

Email ch.ODDB.org Report - Error: oddb.csv - 11/2010

Plugin: ODDB::CsvExportPlugin
Error: NoMethodError
Message: undefined method `de' for nil:NilClass
Backtrace:
(eval):2:in `send'
(eval):2:in `send'
/var/www/oddb.org/src/view/drugs/csv_result.rb:195:in `limitation_text'
/var/www/oddb.org/src/view/drugs/csv_result.rb:315:in `send'
/var/www/oddb.org/src/view/drugs/csv_result.rb:315:in `to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:313:in `collect'
/var/www/oddb.org/src/view/drugs/csv_result.rb:313:in `to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:311:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:311:in `to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:309:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:309:in `to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:330:in `to_csv_file'
/var/www/oddb.org/src/view/drugs/csv_result.rb:330:in `open'
/var/www/oddb.org/src/view/drugs/csv_result.rb:330:in `to_csv_file'
/var/www/oddb.org/src/plugin/csv_export.rb:54:in `_export_drugs'
/var/www/oddb.org/src/plugin/csv_export.rb:24:in `export_drugs'
/var/www/oddb.org/src/util/updater.rb:81:in `export_oddb_csv'
/var/www/oddb.org/src/util/updater.rb:421:in `call'
/var/www/oddb.org/src/util/updater.rb:421:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:79:in `export_oddb_csv'
(eval):1:in `_admin'
/var/www/oddb.org/src/util/failsafe.rb:9:in `call'
/var/www/oddb.org/src/util/failsafe.rb:9:in `failsafe'
/var/www/oddb.org/src/util/oddbapp.rb:1399:in `_admin'
/var/www/oddb.org/src/util/oddbapp.rb:1397:in `initialize'
/var/www/oddb.org/src/util/oddbapp.rb:1397:in `new'
/var/www/oddb.org/src/util/oddbapp.rb:1397:in `_admin'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1515:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1589:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1430:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1347:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `start_service'
/var/www/oddb.org/bin/oddbd:38

Notes

This is the same error as yesterday
I confirm the same result in my local as well
After the reboot of daemon on production server, the export_csv runs correctly ch.ODDB.org Report - oddb.csv - 11/2010
- We guess this is caused by some memory (cache) data problem

Check mail sending process and put some logging code

Normal email sending process

2010-11-01 11:20:10 CET getin log_notify_bsv
2010-11-01 11:21:18 CET getin Log.notify (SL-Update)
2010-11-01 11:21:31 CET return_value_log_notify = [""mhatakeyama@ywesee.com"]
2010-11-01 11:21:31 CET getin Log.notify (SL-Update)
2010-11-01 11:21:54 CET return_value_log2_notify = ["mhatakeyama@ywesee.com"]
2010-11-01 11:21:54 CET getin update_bsv_followers

This time

2010-11-03 04:11:01 CET getin log_notify_bsv
2010-11-03 04:12:05 CET getin Log.notify (SL-Update)
2010-11-03 04:12:17 CET getin Log.notify (SL-Update)
2010-11-03 04:12:17 CET return_value_update_bsv=nil

Notes

'return_value_log_notify' and 'return_value_log2_notify' did not come
Also 'getin Log.notify (SL-Update)' means the process went in src/util/log.rb#notify anyway
This means the process stopped somewhere in the following part of src/util/updater.rb#log_notify_bsv

src/util/log.rb#notify

Notes

There is 'rescue Errno::ENOENT' which means a system call error, no such a file or directory
But nothing does here if the error is caught, just the process stops

Update and test locally

src/util/updater.rb#log_notify_bsv

    def log_notify_bsv(plug, date, subj='SL-Update')

      LogFile.append('oddb/debug', " getin log_notify_bsv", Time.now)
      LogFile.append('oddb/debug', " date=" + date.inspect.to_s, Time.now)

      pointer = Persistence::Pointer.new([:log_group, :bsv_sl], [:log, date])
      LogFile.append('oddb/debug', " after pointer creating", Time.now)
      values = log_info(plug)
      LogFile.append('oddb/debug', " after log_info(plug)", Time.now)
      if log = pointer.resolve(@app)
        change_flags = values[:change_flags]
        if previous = log.change_flags
          previous.each do |ptr, flgs|
            if flags = change_flags[ptr]
              flags.concat flgs
              flags.uniq!
            else
              change_flags[ptr] = flgs
            end
          end
        end
      end
      LogFile.append('oddb/debug', " before @app.update", Time.now)
      log = @app.update(pointer.creator, values)

      LogFile.append('oddb/debug', " after @app.update", Time.now)
      #log.notify(subj)
      return_value_log_notify = log.notify(subj)
      LogFile.append('oddb/debug', " return_value_log_notify = " + return_value_log_notify.inspect.to_s, Time.now)

      log2 = Log.new(date)
      log2.update_values log_info(plug, :log_info_bsv)

      return_value_log2_notify = log2.notify(subj)
      LogFile.append('oddb/debug', " return_value_log2_notify = " + return_value_log2_notify.inspect.to_s, Time.now)
      #log2.notify(subj)
      return_value_log2_notify
    end

src/util/log.rb#notify

    def notify(subject = nil, reply_to = nil)
      LogFile.append('oddb/debug', " getin Log.notify (SL-Update)", Time.now) if subject =~ /SL-Update/

      subj = [
        'ch.ODDB.org Report',
        subject,
        (@date_str || @date.strftime('%m/%Y')),
      ].compact.join(' - ')

      text = text_part(@report)

      parts = @parts.nil? ? [] : @parts.dup
      LogFile.append('oddb/debug', " @files=" + @files.inspect.to_s, Time.now)
      unless(@files.nil?)
        @files.each { |path, (mime, iconv)|
          begin
            content = File.read(path)
            if iconv
              content = Iconv.new(iconv, 'UTF-8').iconv content
            end
            parts.push([mime, File.basename(path), content])
          #rescue Errno::ENOENT
          rescue Errno::ENOENT => e
            LogFile.append('oddb/debug', " " + e.inspect.to_s + "\n" + e.backtrace.inspect.to_s, Time.now)
          end
        }
      end
      LogFile.append('oddb/debug', " start outgoing process", Time.now)
      outgoing = if(parts.empty?)
        text
      else
        multipart = TMail::Mail.new
        multipart.parts << text
        parts.each { |mime, name, content|
          mtype, stype = mime.split('/')
          multipart.parts << file_part(mtype, stype, name, content)
        }
        multipart
      end

      outgoing.from = @mail_from || self::class::MAIL_FROM
      if reply_to
        outgoing.reply_to = reply_to
      end
      @recipients = (@recipients + self::class::MAIL_TO).uniq
      outgoing.subject = subj
      outgoing.date = Time.now
      outgoing['User-Agent'] = 'ODDB Updater'

      LogFile.append('oddb/debug', " before send_mail(outgoing)", Time.now)
      send_mail(outgoing)
    end

Run update_bsv

masa@masa ~/ywesee/oddb.org $ bin/admin 
ch.oddb> Updater.new(self).update_bsv
-> mhatakeyama@ywesee.com
ch.oddb> Updater.new(self).update_bsv
->

Result

(First run)
2010-11-03 09:48:26 CET getin update_bsv
2010-11-03 09:48:30 CET getin BsvXmlPlugin.update
2010-11-03 09:48:30 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2010-11-03 09:48:30 CET save_dir   = /home/masa/ywesee/oddb.org/data/xml
2010-11-03 09:48:30 CET getin download_file
2010-11-03 09:48:34 CET save_file   = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.03.zip
2010-11-03 09:48:34 CET latest_file = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip
2010-11-03 09:48:34 CET File.exists?(/home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = false
2010-11-03 09:48:34 CET path = "/home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.03.zip"
2010-11-03 09:59:08 CET return_value_BsvXmlPlugin.update = "/home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.03.zip"
2010-11-03 09:59:08 CET getin log_notify_bsv
2010-11-03 09:59:08 CET date=Mon, 01 Nov 2010
2010-11-03 09:59:08 CET after pointer creating
2010-11-03 09:59:28 CET after log_info(plug)
2010-11-03 09:59:28 CET before @app.update
2010-11-03 09:59:28 CET after @app.update
2010-11-03 09:59:28 CET getin Log.notify (SL-Update)
2010-11-03 09:59:28 CET @files={}
2010-11-03 09:59:28 CET start outgoing process
2010-11-03 09:59:28 CET before send_mail(outgoing)
2010-11-03 09:59:33 CET return_value_log_notify = ["mhatakeyama@ywesee.com"]
2010-11-03 09:59:33 CET getin Log.notify (SL-Update)
2010-11-03 09:59:33 CET @files={}
2010-11-03 09:59:33 CET start outgoing process
2010-11-03 09:59:33 CET before send_mail(outgoing)
2010-11-03 09:59:37 CET return_value_log2_notify = ["mhatakeyama@ywesee.com"]

(Second run)
2010-11-03 10:00:42 CET getin update_bsv
2010-11-03 10:00:42 CET getin BsvXmlPlugin.update
2010-11-03 10:00:42 CET target_url = http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip
2010-11-03 10:00:42 CET save_dir   = /home/masa/ywesee/oddb.org/data/xml
2010-11-03 10:00:42 CET getin download_file
2010-11-03 10:00:46 CET save_file   = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-2010.11.03.zip
2010-11-03 10:00:46 CET latest_file = /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip
2010-11-03 10:00:46 CET File.exists?(/home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = true
2010-11-03 10:00:47 CET FileUtils.compare_file(/tmp/foo.5521.0, /home/masa/ywesee/oddb.org/data/xml/XMLPublications-latest.zip) = true
2010-11-03 10:00:47 CET path = nil
2010-11-03 10:00:47 CET return_value_BsvXmlPlugin.update = nil

Commit

Added more logging code for checking update_bsv process

Set up Ebps

Git

git clone git://scm.ywesee.com/ebps

Install

masa@masa ~/ywesee/ebps $ cp ../rclconf/setup.rb .
masa@masa ~/ywesee/ebps $ ruby1.9 setup.rb

Commit

Added setup.rb for installation

Install libraries

$ git clone git://scm.ywesee.com/rclconf
$ cd rclconf
$ ruby1.9 setup.rb
$ sudo ruby1.9 /usr/bin/gem install rmail
$ sudo ruby1.9 /usr/bin/gem install rmagick
$ sudo ruby1.9 /usr/bin/gem install builder
$ sudo ruby1.9 /usr/bin/gem install rubyzip
$ ruby1.9 /usr/bin/gem list

*** LOCAL GEMS ***

builder (2.1.2)
minitest (1.6.0)
rake (0.8.7)
rdoc (2.5.8)
rmagick (2.13.1)
rmail (1.0.0)
rubyzip (0.9.4)
search (0.0.1)

Test run

masa@masa ~/ywesee/ebps $ ruby1.9 bin/ebps 
/home/masa/ywesee/ebps/lib/ebps/util/mail.rb:14:in `basename': can't convert nil into String (TypeError)
        from /home/masa/ywesee/ebps/lib/ebps/util/mail.rb:14:in `notify'
        from bin/ebps:85:in `ensure in <main>'
        from bin/ebps:85:in `<main>'

Notes

We need confinguration files and data files to run

Check configuration files

masa@masa ~/ywesee/ebps $ grep -r "etc/ebps" *
lib/ebps/config.rb:    "/etc/ebps/#{run_as}.yml",
lib/ebps/config.rb:    '/etc/ebps/ebps.yml',

lib/ebps/config.rb

  default_config_files = [
    "/etc/ebps/#{run_as}.yml",
    '/etc/ebps/ebps.yml',
  ]

Notes

It is the same structure as the other projects, de.oddb.org, oddb.org etc.

Download the setting and data files

masa@masa ~/ebps_setup/etc/ebps $ scp mhatakeyama@$FAST:/etc/ebps/*.yml .
masa@masa ~/ebps_setup/var $ scp -r mhatakeyama@$FAST:/var/ebps .

Copy the files

masa@masa /etc/ebps $ sudo cp ~/ebps_setup/etc/ebps/*.yml .
masa@masa /etc/ebps $ ls
masa@masa /var $ sudo cp -r ~/ebps_setup/var/ebps .
masa@masa /var $ ls ebps/

Install firefox plugin

http://www.epubread.com/de/

Run bin/ebps

masa@masa ~/ywesee/ebps $ ruby1.9 bin/ebps 
/home/masa/ywesee/ebps/lib/ebps/util/mail.rb:14:in `basename': can't convert nil into String (TypeError)
        from /home/masa/ywesee/ebps/lib/ebps/util/mail.rb:14:in `notify'
        from bin/ebps:85:in `ensure in <main>'
        from bin/ebps:85:in `<main>'

Run

masa@masa ~/work $ ruby1.9 oddb_firefox

Result

Error email Ebook compendium_ch.oddb.org.firefox.epub could not be completed (03.11.2010 11:43:36)

Set up and try again

Set files

file:///etc/ebps/meddrugs.ch_de_firefox.yml -(to open the file locally in firefox use this Add-On: https://addons.mozilla.org/de/firefox/addon/281/ and then open the link via context menu. Also see: http://kb.mozillazine.org/Links_to_local_pages_don%27t_work)

Retry

masa@masa /var/ebps/bin $ sudo ruby1.9 meddrugs.ch_de_firefox

Result

Ebook meddrugs_ch_firefox.epub finished at 03.11.2010 13:44:32

Input file for conversion: /var/ebps/data/yaml/fachinfos.ch.oddb.yaml

Decorate: JustMedical.decorate

The Ebook meddrugs_ch_firefox.epub was generated and stored in /var/ebps/data/books

Update the decorator of meddrugs.ch_de_firefox

New task

Create a link for FI (fachinfo) book to go directly to the search result of "Markenname" in oddb.org

Make a link on title

Click result

Refer

Email Just-Medical Links for FI and PI

Memo

For local test, we can also do as follows:

masa@masa ~/work $ ruby1.9 meddrugs.ch_de_firefox config="meddrugs.ch_de_firefox.yml"

file:///home/masa/ebps_setup/local_files/etc/ebps/meddrugs.ch_de_firefox.yml

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101103-decorators-for-ebooks

Check update_bsv log last night

Export_csv fails on production server

Check mail sending process and put some logging code

Set up Ebps

Set up and try again

Update the decorator of meddrugs.ch_de_firefox

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101103-decorators-for-ebooks

Check update_bsv log last night

Export_csv fails on production server

Check mail sending process and put some logging code

Set up Ebps

Set up and try again

Update the decorator of meddrugs.ch_de_firefox

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler