view · edit · sidebar · attach · print · history

20101117-create-grant_download-command-de_oddb_org

<< Masa.20101119-create-grant_download-command-de_oddb_org | 2010 | Masa.20101117-create-grant_download-command-de_oddb_org >>

Regarding a log of grant_download
Review yesterday
Implement GrantDownload class
Implement grant_download method for client
Interim Summary
Check logging and reporting processes in oddb.org

Goal

Create grant_download / 90%

Milestones

BraSt: Should Log be saved in ODBA cache?
~~Implement GrantDwonload class~~ 14:00
~~Implement grant_download method for bin/admin~~
~~Implement grant_download method for client~~ 14:40
Logging process
Test case
Help command
etc.

Memo

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> ODDB::Business::GrantDownload.all
-> aa@bb.ccaaa@bbb.cccaa@bb.ccc
de.oddb> ODDB::Business::GrantDownload.all.inspect
-> String
de.oddb> ODDB::Business::GrantDownload.all.join(" ")
-> aa@bb.ccc aaa@bbb.ccc aa@bb.cc
de.oddb> ODDB::Business::GrantDownload.find_by_email 'aa@bb.ccc'
-> aa@bb.ccc
de.oddb> exit
-> Goodbye

hash sort http://tobysoft.net/wiki/index.php?Ruby%2FHash%A4%C7%A5%BD%A1%BC%A5%C8%A4%B9%A4%EB%CA%FD%CB%A1 (japanese)
what is '@session.is_crawler?' in oddb.org/src/state/global.rb#download

Summary

http://masay.ch/2010/11/18/create-grant_download-command-de_oddb_org-3/

Commits
ToDo Tomorrow

Logging and reporting processes
grant_download (bin/admin) Help
Test case

Keep in Mind

A bug import_gkv Tue Nov 16 02:00:10 2010: de.oddb.org Zubef (PDF)
On Ice
emerge --sync

Regarding a log of grant_download

Should Log be saved in ODBA cache?

It is easy to save the log with the email address (odba_index) on the cache
It is easy too to search the download history
But the log will increase in the cache (memory), though it is small data.
oddb.org outputs a downloading log in the flat data file not in the memory
In stead of that, it costs a little bit to search the downloading history of a user

Davatz-san's mail

Download Report Email
Fwd: ch.ODDB.org Report - Download-Statistics - 11/2010
Every sunday the following download log will come by email
The downloading by grant_download command should be included in this log

2010-11-02 11:55:00
CET;11.22.33..44;aaa@bbb.ccc;../data/downloads/oddb.csv
2010-11-02 12:00:00
CET;11.22.33..44;aaa@bbb.ccc;../data/downloads/oddb.csv
2010-11-02 13:13:00
CET;11.22.33..44;aaa@bbb.ccc;../data/downloads/oddb2.csv
2010-11-03 09:09:00
CET;11.22.33..44;aaa@bbb.ccc;../data/downloads/oddb.csv
2010-11-12 07:07:00
CET;11.22.33..44;aaa@bbb.ccc;../data/downloads/oddb.yaml

(day and time)
CET;(ip address);(email address);(file path)

Review yesterday

1. grant_download method for client lib/oddb/html/state/global.rb

  def grant_download
    _download(@session.user_input(:file))
  end

2. Create a new event lib/oddb/html/util/validator.rb

  EVENTS = [ :ajax_autofill, :ajax_create_active_agent,
...
             #:sequence, :sort, :update ]
             :sequence, :sort, :update, :grant_download ]

Test

Reboot de.oddb.org/bin/oddbd
Access to http://de.oddb.masa.org/xx/xxx/grant_download/file/de.oddb.csv
Result: success

2. GrantDownload class lib/oddb/business/grant_download.rb

#!/usr/bin/env ruby
# ODDB::Business::GrantDownload -- de.oddb.org -- 18.11.2010 -- mhatakeyama@ywesee.com

require 'oddb/model'

module ODDB
  module Business
    class GrantDownload < Model
      multilingual :email
      def initialize(email)
        self.email.de = email
      end
    end
  end
end

Modify grant_download method for client lib/oddb/html/state/global.rb

require 'oddb/business/grant_download'
...
  def grant_download
    if(user = ODDB::Business::GrantDownload.find_by_uid(@session.user_input(:id)))
      p "true"
    else
      p "false"
    end
    _download(@session.user_input(:file))
  end

Test

Reboot de.oddb.org/bin/oddbd
Access to http://de.oddb.masa.org/xx/xxx/grant_download/file/de.oddb.csv/uid/123
Result: 'false' => find_by_uid method is recognized but it cannot find the corresponding data from the cache

3. grant_download method for bin/admin lib/oddb/util/server.rb

      def grant_download(uid)
        if(user = ODDB::Business::GrantDownload.find_by_uid(uid))
          user
        else
          'failure'
        end
      end

Test

Reboot de.oddb.org/bin/oddbd

Result

masa@masa ~/ywesee/de.oddb.org $ bin/admin 
de.oddb> grant_download 123
-> failure
de.oddb>

Notes

GrantDownload class is recognized and find_by_uid method worked

4. find_by_email method

lib/oddb/persistence/odba.rb

require 'oddb/persistence/odba/business/grant_download'

lib/oddb/persistence/odba/business/grant_download.rb

!/usr/bin/env ruby
# ODDB::Business::GrantDownload -- de.oddb.org -- 18.11.2010 -- mhatakeyama@ywesee.com

require 'oddb/business/grant_download'
require 'oddb/persistence/odba/model'

module ODDB
  module Business
    class GrantDownload < Model
      odba_index :email
    end
  end
end

lib/oddb/util/server.rb

      def grant_download(arg)
        if arg =~ /\d+/ or arg.kind_of? Integer
          if(user = ODDB::Business::GrantDownload.find_by_uid(arg))
            p 'find_by_uid success'
            p user
            user.uid
          else
            'find_by_uid failure'
          end
        elsif arg =~ /\@/
          if(user = ODDB::Business::GrantDownload.find_by_email(arg))
            p 'find_by_email success'
            p user
            user.uid
          else
            'find_by_email failure'
          end
        end
      end

Test

Reboot de.oddb.org/bin/oddbd

Result

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> grant_download 3659983
-> 3659983
de.oddb> grant_download 'aa@bb.cc'
-> 3659983
de.oddb> grant_download '3659983'
-> 3659983
de.oddb> grant_download 123
-> find_by_uid failure
de.oddb> grant_download 'aaa@bbb.ccc'
-> find_by_email failure
de.oddb>

Notes

In order to enable 'find_by_email', the GrantDownload class must be re-defined under lib/oddb/persistence/odba/business directory. I do not know why, though.

Implement GrantDownload class

BraSt

At the first step, I do not care about logging and invoice
There may be a case in which more than two grant is added to the same user
'grant' data should be 1. file 2. expiry_time
The index key will be email to identify the user
grant_download command (bin/admin) will be
- grant_download 'email@domain.com', 'file name', Time.local(yyyy,mm,dd)

Experiment

lib/oddb/business/grant_download.rb

#!/usr/bin/env ruby
# ODDB::Business::GrantDownload -- de.oddb.org -- 18.11.2010 -- mhatakeyama@ywesee.com

require 'oddb/model'

module ODDB
  module Business
    class GrantDownload < Model
      multilingual :email
      attr_reader :grant_list
      def initialize(email)
        self.email.de = email
        @grant_list = {}
      end
      def grant_download(file, expiry_time)
        @grant_list[file] = expiry_time
      end
      def expired?(file)
        if expiry_time = @grant_list[file]
          expiry_time < Time.now
        else
          true
        end
      end
    end
  end
end

lib/oddb/util/server.rb

      def grant_download(*arg)
        if arg.length == 3      # register
          unless user = ODDB::Business::GrantDownload.find_by_email(arg[0])
            p "cannot find the user"
            user = ODDB::Business::GrantDownload.new(arg[0])
            p "created a new user"
          end
          p user
          user.grant_download(arg[1], arg[2])
          user.save
          user.uid
        elsif arg.length == 1   # search
          if user = ODDB::Business::GrantDownload.find_by_email(arg[0])
            p "find the user"
            p user
            user.grant_list
          else
            'There is no registration for ' + arg[0]
          end
        else
          'help'
        end

Reboot de.oddb.org/bin/oddbd

Result

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> grant_download 'aa@bb.cc', 'test.dat', Time.local(2011,1,1)
-> 3659985
de.oddb> grant_download 'aa@bb.cc'
-> test.datSat Jan 01 00:00:00 +0100 2011
de.oddb> exit
-> Goodbye

oddbd log

"cannot find the user"
"created a new user"
#<ODDB::Business::GrantDownload:0x7f7abeefa818 @grant_list={}, @email=#<ODDB::Util::Multilingual:0x7f7abeefa6d8 @synonyms=[], @canonical={:de=>"aa@bb.cc"}>>
"find the user"
#<ODDB::Business::GrantDownload:0x7f7abeefa818 @odba_id=3659985, @odba_persistent=true, @grant_list={"test.dat"=>Sat Jan 01 00:00:00 +0100 2011}, @email=#<ODDB::Util::Multilingual:0x7f7abeefa6d8 @synonyms=[], @canonical={:de=>"aa@bb.cc"}>, @odba_observers=[]>

Notes

Looks good BUT

Reboot de.oddb.org/bin/oddbd again

Result

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb>  grant_download 'aa@bb.cc'
-> 
de.oddb>

oddbd log

"find the user"
#<ODDB::Business::GrantDownload:0x7f611fa07b48 @odba_id=3659985, @odba_persistent=true, @grant_list=#<ODBA::Stub:70027559582800#3659986 @odba_class=Hash @odba_container=70027559583140#3659985>, @email=#<ODDB::Util::Multilingual:0x7f611fa07a30 @synonyms=[], @canonical={:de=>"aa@bb.cc"}>, @odba_observers=[]>
ODBA::Stub was unable to replace Hash#3659986 from ODDB::Business::GrantDownload:#3659985

Notes

ODBA::Stub was unable to replace Hash#3659986 from ODDB::Business::GrantDownload:#3659985
I cannot judge if the file and expiry_time data in the database or not

Experiment

lib/oddb/persistence/odba/business/grant_download.rb

#!/usr/bin/env ruby
# ODDB::Business::GrantDownload -- de.oddb.org -- 18.11.2010 -- mhatakeyama@ywesee.com

require 'oddb/business/grant_download'
require 'oddb/persistence/odba/model'

module ODDB
  module Business
    class GrantDownload < Model
      odba_index :email
      serialize :grant_list
    end
  end
end

Reboot de.oddb.org/bin/oddbd

Result

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> grant_download 'aa@bb.cc', 'test.dat', Time.local(2011,1,1)
-> 3659985
de.oddb> grant_download 'aa@bb.cc'
-> test.datSat Jan 01 00:00:00 +0100 2011
de.oddb> exit
-> Goodbye

oddbd log

"cannot find the user"
"created a new user"
#<ODDB::Business::GrantDownload:0x7fed91c5cd90 @grant_list={}, @email=#<ODDB::Util::Multilingual:0x7fed91c5cc50 @synonyms=[], @canonical={:de=>"aa@bb.cc"}>>
"find the user"
#<ODDB::Business::GrantDownload:0x7fed91c5cd90 @odba_id=3659985, @odba_persistent=true, @grant_list={"test.dat"=>Sat Jan 01 00:00:00 +0100 2011}, @email=#<ODDB::Util::Multilingual:0x7fed91c5cc50 @synonyms=[], @canonical={:de=>"aa@bb.cc"}>, @odba_observers=[]>

Reboot de.oddb.org/bin/oddbd again

Result

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> grant_download 'aa@bb.cc'
-> test.datSat Jan 01 00:00:00 +0100 2011
de.oddb> exit
-> Goodbye

oddbd log

"find the user"
#<ODDB::Business::GrantDownload:0x7ff3ff1074f8 @odba_id=3659985, @odba_persistent=true, @grant_list={"test.dat"=>Sat Jan 01 00:00:00 +0100 2011}, @email=#<ODDB::Util::Multilingual:0x7ff3ff107408 @synonyms=[], @canonical={:de=>"aa@bb.cc"}>, @odba_observers=[]>

Notes

Success!!
Key point is serialize :grant_list

Implement grant_download method for client

lib/oddb/html/state/global.rb

  def grant_download

    if user = ODDB::Business::GrantDownload.find_by_uid(@session.user_input(:uid))
      if user.expired?(@session.user_input(:file))
        _download(@session.user_input(:file))
      end
    end
  end

Check entry

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> grant_download 'aa@bb.cc'
-> test.txtWed Feb 02 00:00:00 +0100 2011test.datSat Jan 01 00:00:00 +0100 2011
de.oddb> exit
-> Goodbye

Reboot de.oddb.org/bin/oddbd

Test case 1: success case

Access to http://de.oddb.masa.org/xx/xxx/grant_download/uid/3659985/file/test.dat

Result

success

Test case 2: failure case, there is no target file

masa@masa ~/ywesee/de.oddb.org $ rm var/downloads/test.dat

Access to http://de.oddb.masa.org/xx/xxx/grant_download/uid/3659985/file/test.dat

Result

failure, and go to top page

Test case 3: failure case, there is the target file but no granted

masa@masa ~/ywesee/de.oddb.org $ cat var/downloads/ttt.dat 
hello

Access to http://de.oddb.masa.org/xx/xxx/grant_download/uid/3659985/file/ttt.dat

Result

failure

Test case 4: failure case, there is the target but expired

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> grant_download 'aa@bb.cc', 'test.txt', Time.local(2010,1,1)
-> 3659985
de.oddb> grant_download 'aa@bb.cc'
-> test.txtFri Jan 01 00:00:00 +0100 2010test.datSat Jan 01 00:00:00 +0100 2011
de.oddb> exit
-> Goodbye

Access to http://de.oddb.masa.org/xx/xxx/grant_download/uid/3659985/file/test.txt

Result

failure

Test case 5: failure case, wrong URL

Result

failure, but the ODBA::OdbaError happens

error in SBSM::Session#process: /xx/xxx/grant_download/
ODBA::OdbaError
Unknown odba_id 
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:626:in `restore_object'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:589:in `load_object'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:232:in `fetch'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:330:in `call'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:330:in `fetch_or_do'

Update

lib/oddb/html/state/global.rb

  def grant_download
    email = @session.user_input(:email)
    file = @session.user_input(:file)
    if email && file && user = ODDB::Business::GrantDownload.find_by_email(email)
      p user
      unless user.expired?(@session.user_input(:file))
        p "not expired"
        _download(@session.user_input(:file))
      end
    end
  end

Result

Everything fails, and without Error

Notes

Instead of find_by_uid, I used find_by_email

Interim Summary

lib/oddb/business/grant_download.rb

#!/usr/bin/env ruby
# ODDB::Business::GrantDownload -- de.oddb.org -- 18.11.2010 -- mhatakeyama@ywesee.com

require 'oddb/model'

module ODDB
  module Business
    class GrantDownload < Model
      multilingual :email
      attr_reader :grant_list
      def initialize(email)
        self.email.de = email
        @grant_list = {}
      end
      def grant_download(file, expiry_time)
        @grant_list[file] = expiry_time
      end
      def expired?(file)
        if expiry_time = @grant_list[file]
          expiry_time < Time.now
        else
          true
        end
      end
    end
  end
end

lib/oddb/persistence/odba.rb

require 'oddb/persistence/odba/business/grant_download'

lib/oddb/persistence/odba/business/grant_download.rb

#!/usr/bin/env ruby
# ODDB::Business::GrantDownload -- de.oddb.org -- 18.11.2010 -- mhatakeyama@ywesee.com

require 'oddb/business/grant_download'
require 'oddb/persistence/odba/model'

module ODDB
  module Business
    class GrantDownload < Model
      odba_index :email
      serialize :grant_list
    end
  end
end

lib/oddb/html/state/global.rb

  def grant_download
    email = @session.user_input(:email)
    file = @session.user_input(:file)
    if email && file && user = ODDB::Business::GrantDownload.find_by_email(email)
      unless user.expired?(@session.user_input(:file))
        _download(@session.user_input(:file))
      end
    end
  end

lib/oddb/util/server.rb

      def grant_download(*arg)
        email = arg[0]
        file  = arg[1]
        expiry_time = arg[2]
        if arg.length == 3      # register
          unless user = ODDB::Business::GrantDownload.find_by_email(email)
            user = ODDB::Business::GrantDownload.new(email)
          end
          user.grant_download(file, expiry_time)
          user.save
          ODDB.config.http_server + '/de/temp/grant_download/email/' + email + '/file/' + file
        elsif arg.length == 1   # search
          if user = ODDB::Business::GrantDownload.find_by_email(email)
            grant_list = user.grant_list.sort_by{|filename, expirytime| expirytime}.reverse
            str = grant_list[0..3].map{|x| x[1].strftime("%Y%m%d") + ', ' + x[0]}.join("\n")
            "grant list(total:" + grant_list.length.to_s + "): " + email + "\n" + str
          else
            'No registration for ' + email
          end
        else
          'help'
        end

Notes

Not yet commit
In order to enable grant_download in bin/admin, reboot oddbd

Logging
Help
Delete function
Test case

Check logging and reporting processes in oddb.org

The logging in oddb.org is done in the following class

src/view/user/download.rb

#!/usr/bin/env ruby
# View::User::Download -- ODDB -- 29.10.2003 -- hwyss@ywesee.com

require 'htmlgrid/passthru'
require 'util/logfile'
require 'plugin/yaml'

module ODDB
    module View
        module User
class Download < HtmlGrid::PassThru
    def init
        if(filename = @session.user_input(:filename))
            @path = File.join('..', 'data', 'downloads', filename)
        end
    end
    def to_html(context)
        line = [
            nil,
            @session.remote_addr,
            @session.user_input(:email),
            @path,
        ].join(';')
        LogFile.append(:download, line, Time.now)
        @session.passthru(@path)
        ''
    end
end
        end
    end
end

The reporting process runs from exporter

src/util/exporter.rb

    def run
...
      run_on_weekday(0) {
        mail_download_stats
        mail_feedback_stats
        #mail_notification_stats
      }
...
      def mail_download_stats
      safe_export 'Mail Download-Statistics' do
        mail_stats('download')
      end
...
        def mail_stats(key)
            date = @@today
            if(date.mday < 8)
                date = date << 1
            end
            log = Log.new(date)
            begin
                log.report = File.read(LogFile.filename(key, date))
            rescue StandardError => e
                log.report = ([
                    "Nothing to Report.",
                    nil,
                    e.class,
                    e.message
                ] + e.backtrace).join("\n")
            end
            log.notify("#{key.capitalize}-Statistics")
        end

Notes

This method is called once a week
Just read the flat file data and send it by email
The logging process just writes the text data into the flat text file

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101117-create-grant_download-command-de_oddb_org

Regarding a log of grant_download

Review yesterday

Implement GrantDownload class

Implement grant_download method for client

Test case 1: success case

Test case 2: failure case, there is no target file

Test case 3: failure case, there is the target file but no granted

Test case 4: failure case, there is the target but expired

Test case 5: failure case, wrong URL

Interim Summary

Check logging and reporting processes in oddb.org

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20101117-create-grant_download-command-de_oddb_org

Regarding a log of grant_download

Review yesterday

Implement GrantDownload class

Implement grant_download method for client

Test case 1: success case

Test case 2: failure case, there is no target file

Test case 3: failure case, there is the target file but no granted

Test case 4: failure case, there is the target but expired

Test case 5: failure case, wrong URL

Interim Summary

Check logging and reporting processes in oddb.org

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler