view · edit · sidebar · attach · print · history

20101119-create-grant_download-command-de_oddb_org

<< Masa.20101122-testcase-grant_download-command | 2010 | Masa.20101118-create-grant_download-command-de_oddb_org >>


  1. Check logging process in both oddb.org (ch) and de.oddb.org
  2. Check how to use Logger class
  3. Design Logging and Reporting process
  4. Define logging process
  5. Define reporting process
  6. Update grant_download command help
  7. Define compression process

Goal
  • Create grant_download / 90%
Milestones
  1. delete command suspend
  2. Logging process
  3. Reporting process
  4. grant_download help command 16:45
  5. Compression process refer to lib/oddb/export/server.rb
  6. Test cases (grant_download, Logging, Reporting)
  7. etc.
Summary
Commits
ToDo Tomorrow
  • Test cases or import_gkv bug
Keep in Mind
  1. A bug import_gkv Tue Nov 16 02:00:10 2010: de.oddb.org Zubef (PDF)
  2. Log Error: on production server, de.oddb.org/log/import_dimdi, import_pharmnet
  3. On Ice
  4. emerge --sync

How to use grant_download command

Commit

registration

grant_download 'mhatakeyama@ywesee.com', 'test.dat', Time.local(2010,12,31)

show list

grant_download 'mhatakeyama@ywesee.com'

Sample log

masa@masa ~/ywesee/de.oddb.org $ bin/admin 
de.oddb> grant_download 'mhatakeyama@ywesee.com'
-> No registration for mhatakeyama@ywesee.com

de.oddb> grant_download 'mhatakeyama@ywesee.com', 'test.dat', Time.local(2010,12,31)
-> http://de.oddb.masa.org/de/temp/grant_download/email/mhatakeyama@ywesee.com/file/test.dat

de.oddb> grant_download 'mhatakeyama@ywesee.com'
-> grant list(total:1): mhatakeyama@ywesee.com
20101231, test.dat

de.oddb> grant_download 'mhatakeyama@ywesee.com', 'test.dat', Time.local(2011,12,31)
-> http://de.oddb.masa.org/de/temp/grant_download/email/mhatakeyama@ywesee.com/file/test.dat

de.oddb> grant_download 'mhatakeyama@ywesee.com'
-> grant list(total:1): mhatakeyama@ywesee.com
20111231, test.dat

de.oddb> exit
-> Goodbye

Notes

  • The grant list shows up the top 4 entries which is sorted by date
  • There is no 'delete' command at the moment.
  • But we can delete an entry by using the following primitive method
de.oddb> ODDB::Business::GrantDownload.find_by_email('mhatakeyama@ywesee.com').delete
-> mhatakeyama@ywesee.com
de.oddb> grant_download 'mhatakeyama@ywesee.com'
-> No registration for mhatakeyama@ywesee.com
  • If we know the odba_id of an entry in the cache, 'find_by_uid' method is also available to find a object in the cache
  • But this is only for de.oddb.org, not for all ODBA system.
  • We can see the other methods than 'delete' for an ODBA cache object in lib/oddb/persistence/odba/model.rb, ODBA class methods is also defined here.
  • We can see the other methods than 'find_by*' for an ODBA class in odba/lib/odba/persistable.rb

Memo

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> ODDB::Business::GrantDownload.all
-> aa@bb.ccaaa@bbb.cccaa@bb.ccc
de.oddb> ODDB::Business::GrantDownload.all.inspect
-> String
de.oddb> ODDB::Business::GrantDownload.all.join(" ")
-> aa@bb.ccc aaa@bbb.ccc aa@bb.cc
de.oddb> ODDB::Business::GrantDownload.find_by_email 'aa@bb.ccc'
-> aa@bb.ccc
de.oddb> ODDB::Business::GrantDownload.search_by_email 'aa@'
-> aa@bb.ccaa@bb.ccc
de.oddb> exit
-> Goodbye

Question

  • what is '@session.is_crawler?' in oddb.org/src/state/global.rb#download

Check logging process in both oddb.org (ch) and de.oddb.org

Davatz-san's mail

Both ch.oddb and de.oddb are both put into the same mail. It is always sent on Sunday. 

Hypothesis

  • I guess
    • oddb.org (src/util/exporter.rb#mail_stats) in charge of sending an email report
    • de.oddb.org does only outputting the log in the same place (file) as oddb.org does.

Logging and reporting processes in oddb.org http://dev.ywesee.com/wiki.php/Masa/20101118-create-grantDownload-command-deOddbOrg#CheckLogging

Logger class in de.oddb.org lib/oddb/util/logger.rb

Check log directory on production server

de.oddb.org/log $ ls
import_dimdi  import_gkv  import_pharma24  import_pharmnet  import_whocc  mm  oddb

Notes

      def Updater.run(today = Date.today)
        run_logged_job 'import_dimdi'
        run_logged_job 'import_gkv'
        case today.day
        when 1
          run_logged_job 'import_pharmnet'
          run_logged_job 'import_whocc'
        when 15
          run_logged_job 'import_pharma24'
        end
      end
      def Updater.run_logged_job job
        dir = ODDB.config.oddb_dir
        cmd = File.join dir, 'jobs', job
        log = File.join dir, 'log', job
        IO.popen "#{cmd} log_file=#{log}" do |io|
          # wait for importer to exit
        end
      end

Notes

  • The value of 'log_file=' option becomes ODDB.config.log_file in the source code
  • This is reflected in lib/oddb/util/logger.rb
  • In the block of 'run_logged_job' method, the log file is saved in the different place from the initial setting of ODDB.config (etc/oddb.yml, lib/oddb/config.rb)

Report sending process in de.oddb.org is concentrated in lib/oddb/util/mail.rb

  • In particular, Mail.notify_invoice (when :download) is suspect
        invoice.items.each { |item|
          case item.type
          when :download
            unless instructed
              parts.push lnf.lookup(:download_mail_body)
              parts.push lnf.lookup(:download_mail_instr)
            end
            instructed = true
            parts.push lnf._event_url(:collect,
                                      [:invoice, invoice.id,
                                       :file, Download.compressed_download(item)])
...

Hypotheses

  • I guess de.oddb.org does not output the downloading log in a file
  • In de.oddb.org, a report mail is sent without logging when the downloading is done

Check how to use Logger class

References

Design Logging and Reporting process

BraSt

  • lib/oddb/util/logger.rb should be updated, otherwise the log will be outputted into ODDB.config.log_file
  • The report sending process can be delegated to Mail.notify_admins (be careful of the format)
  • But I have to check how to run the job once a week
  • There is no 'run_on_weekday' method definition in de.oddb.org. Should I define it as well as oddb.org?
  • In this time, just focus on the downloading by grant_download method
  • The logging process should be in ODDB::View::Download#to_html as well as oddb.org
  • Check how to lotate the log file (japanese) tutorial logger http://jp.rubyist.net/magazine/?0008-BundledLibraries
  • Check how to compress the log file (Japanese) tutorial zlib http://jp.rubyist.net/magazine/?0018-BundledLibraries

Define logging process

Point

  • The current processes must run as well without modification
  • To set the log file name and directory
  • The easiest way is to define a new class besides the lib/oddb/util/logger.rb
  • Log rotation is once in a month
  • When a report mail is sent, it checks the rotated log file in the log directory and compresses it

Experiment

lib/oddb/html/view/download.rb

require 'htmlgrid/passthru'

module ODDB
  module Html
    module View
class Download < HtmlGrid::PassThru
  def to_html(context)
    line = [
      Time.now.strftime('%Y-%m-%d %H:%M:%S %Z'),
      @session.remote_addr,
      @session.user_input(:email),
      @path,
    ].join(';')
    Logger.new('/home/masa/work/log.dat', 'monthly').info line
    @session.passthru(@model)
    ''
  end
end
    end
  end
end

Check grant

de.oddb> grant_download 'mhatakeyama@ywesee.com', 'test.dat', Time.local(2011,1,1)
-> http://de.oddb.masa.org/de/temp/grant_download/email/mhatakeyama@ywesee.com/file/test.dat
de.oddb> grant_download 'mhatakeyama@ywesee.com'
-> grant list(total:1): odba_id: 3659988
20110101, test.dat

Reboot de.oddb.org/bin/oddbd

Access to http://de.oddb.masa.org/de/temp/grant_download/email/mhatakeyama@ywesee.com/file/test.dat

Result

masa@masa ~/ywesee/de.oddb.org $ cat /home/masa/work/log.dat 
# Logfile created on Fri Nov 19 13:52:48 +0100 2010 by logger.rb/22283
2010-11-19 13:59:55 CET;127.0.0.1;mhatakeyama@ywesee.com;/home/masa/ywesee/de.oddb.org/var/downloads/test.dat

Experiment

lib/oddb/config.rb

    'download_log_dir'      => File.expand_path('log/download', oddb_dir),

lib/oddb/html/view/download.rb

require 'htmlgrid/passthru'
require 'fileutils'

module ODDB
  module Html
    module View
class Download < HtmlGrid::PassThru
  def to_html(context)
    if download_file = @session.user_input(:file) and email = @session.user_input(:email)
      # Logging a file download 
      # Normally it is saved in log/download directory
      time = Time.now
      log_dir = File.join(ODDB.config.download_log_dir, time.year.to_s)
      FileUtils.mkdir_p log_dir
      log_file = File.join(log_dir, time.month.to_s + '.log')
      Logger.new(log_file).info do
        [
          time.strftime('%Y-%m-%d %H:%M:%S %Z'),
          @session.remote_addr,
          email,
          File.join(ODDB.config.export_dir, download_file)
        ].join(';')
      end
    end

    @session.passthru(@model)
    ''
  end
end
    end
  end
end

Check directory

masa@masa ~/ywesee/de.oddb.org $ ls 
bin  data  doc  etc  jobs  lib  LICENSE  README  test  var

Note

  • There is no 'log' directory

Access to http://de.oddb.masa.org/de/temp/grant_download/email/mhatakeyama@ywesee.com/file/test.dat

Result

masa@masa ~/ywesee/de.oddb.org $ cat log/download/2010/11.log 
# Logfile created on Fri Nov 19 14:31:36 +0100 2010 by logger.rb/22283
2010-11-19 14:31:36 CET;127.0.0.1;mhatakeyama@ywesee.com;/home/masa/ywesee/de.oddb.org/var/downloads/test.dat

Notes

  • Should I define a new class for this?
    • because it is not easy to read the code of 'to_html'

Commit

Next

  • Compress log files later
  • Reporting process

Define reporting process

Experiment

lib/oddb/export/server.rb

      def Server.report_download
        # Note: if the day is in new month from the last week,
        # it may not include all the log information.
        subject = sprintf("de.ODDB.org Report - Download-Statistics - %s",
                          Time.now.strftime('%m/%Y'))
        time = Time.now
        log_dir = File.join(ODDB.config.download_log_dir, time.year.to_s)
        log_file = File.join(log_dir, time.month.to_s + '.log')
        begin
          lines = File.readlines(log_file)
        rescue StandardError => e
          lines = [
             "Nothing to Report.",
             nil,
             e.class,
             e.message
          ] + e.backtrace
        end
        Util::Mail.notify_admins(subject, lines)
      end

      def Server.run(today = Date.today)
        on_monthday(1, today) {
          export_chde_xls
        }
        on_monthday(2, today) do
          export_info_yaml :fachinfo
          export_info_yaml :patinfo
        end
        on_weekday(0, today) do
          report_download
        end
      end

      def Server.on_weekday(wday, today = Date.today, &block)
        # 0-6, 0:Sunday
        if(today.wday == wday)
          block.call
        end
      end

Run

masa@masa ~/ywesee/de.oddb.org $ bin/admin 
de.oddb> ODDB::Exporter.report_download
-> mhatakeyama@ywesee.com
de.oddb> exit
-> Goodbye

Result (if there is a log file)

# Logfile created on Fri Nov 19 15:45:03 +0100 2010 by logger.rb/22283

2010-11-19 15:45:03 CET;127.0.0.1;mhatakeyama@ywesee.com;/home/masa/ywesee/de.oddb.org/var/downloads/test.dat

Result (if there is no log file)

Nothing to Report.

Errno::ENOENT
No such file or directory - /home/masa/ywesee/de.oddb.org/log/download/2010/11.log
/home/masa/ywesee/de.oddb.org/lib/oddb/export/server.rb:46:in `readlines'
/home/masa/ywesee/de.oddb.org/lib/oddb/export/server.rb:46:in `report_download'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1516:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1590:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1431:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1348:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `start_service'
bin/exportd:35

Notes

  • Looks good

Commit

su
svc -d /service/de.oddb-exporter
svc -u /service/de.oddb-exporter
svstat /service/de.oddb-exporter

svc -d /service/de.oddb
svc -u /service/de.oddb
svstat /service/de.oddb
  • Create directory download and change the owner, otherwise Permission error happens because the system creates a directory and file
cd /var/www/de.oddb.org/log
mkdir download
su
chown apache download

Update grant_download command help

lib/oddb/util/server.rb#grant_download

Commit

Result

masa@masa ~/ywesee/de.oddb.org $ bin/admin 
de.oddb> grant_download
-> Usage:
  Set  grant: grant_download 'email address', 'file', Time.local(20yy,mm,dd)
  Show grant: grant_download 'email address'

Define compression process

Plan

  • lib/oddb/export/server.rb
  • on_yearday method
  • use Server.compress method
view · edit · sidebar · attach · print · history
Page last modified on July 13, 2011, at 11:54 AM