20101115-create-grant_download-command-de_oddb_org

<< Masa.20101116-create-grant_download-command-de_oddb_org | 2010 | Masa.20101111-setup-eshop update-customer-data >>

---

1. Function of grant_download command
2. Review
3. Study grant_download oddb.org
4. Check de.oddb.org structure (View and State classes)
5. Experiments in de.oddb.org

---

Goal

• Create grant_download / 50%

Milestones

1. Review 8:15
2. BraSt 8:30
3. Study a method to link a file
4. Study a method to expire a link 15:20
5. Comparison to de.oddb.org (View and State classes) 16:20

Summary

• http://masay.ch/2010/11/15/create-grant_download-command-de-oddb-org/

Commits

ToDo Tomorrow

• Test a new method in de.oddb.org

Keep in Mind

• On Ice
• emerge --sync

---

Function of grant_download command

Davatz-san's mail

• For a start we need to set the file that is being downloaded,
• the email of the user and the end date (DDMMYYYY) after which you can not download the file anymore.
• The command should also be able to show all old and active dowloads by email address of the user.

Review

oddb.org

• bin/admin calls functions in src/util/oddbapp.rb
• grant_download is defined in src/util/oddbapp.rb

de.oddb.org

• bin/admin calls the functions in lib/oddb/util/server.rb
• grant_download command should be also in lib/oddb/util/server.rb

BraSt

• I have to understand more about the translation mechanism of url (SBSM and HtmlGrid)
  • and think how to link to a file downloaded
  • It may be good way to study it from oddb.org source code (grant_download method)
• I have to understand how to expire a download link, too

Study grant_download oddb.org

Experiment

masa@masa ~/ywesee/oddb.org $ bin/admin 
ch.oddb> grant_download 'mhatakeyama@ywesee.com', 'oddb.csv', 1, Time.local(2011,1,1)
-> http://oddb.masa.org/de/gcc/download/invoice/25518091/email/mhatakeyama@ywesee.com/filename/oddb.csv

masa@masa ~/ywesee/oddb.org/data/downloads $ ls
masa@masa ~/ywesee/oddb.org/data/downloads $ 

Access to the link on browser

• 'Es tut uns leid' error

Experiment

masa@masa ~/ywesee/oddb.org/data/downloads $ ls -al
insgesamt 5444
drwxr-xr-x 2 masa masa       8 15. Nov 09:10 .
drwxr-xr-x 8 masa masa      48 15. Nov 09:03 ..
-rw-r--r-- 1 masa masa 5573371 15. Nov 09:10 oddb.csv

Access to the same link

• Succeeded in downloading the file

Experiment

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> grant_download 'aaa@bbb.ccc', 'oddb.csv', 1000, Time.local(2011,2,2)
-> http://oddb.masa.org/de/gcc/download/invoice/25518094/email/aaa@bbb.ccc/filename/oddb.csv

Access to the link

• Succeeded in downloading the same file

Notes

• All the URLs are liked to the same file in data/downloads directory

Hypotheses

• The logic may be switched depending on the URL string
• The URL string is translated anyhow into the actual path
• The id (number) and email address may be refered to in the database for checking the expiration

Experiment

/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb#process

        def process(request)
            begin
                @request = request
        @request_method = request.request_method
        @request_path = @request.unparsed_uri
open("/home/masa/work/test.dat","a"){|f| f.print "@request_path=", @request_path, "\n"}
...

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

masa@masa ~/work $ cat test.dat 
@request_path=/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat

Notes

• The URL string is caught in SBSM::Session#process method

Experiment

/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb#import_user_input

    def import_user_input(request)
open("/home/masa/work/test.dat","a"){|f| f.print "getin import_user_input\n"}
            # attempting to read the cgi-params more than once results in a
            # DRbConnectionRefused Exception. Therefore, do it only once...
            return if(@user_input_imported)
      request.params.each { |key, value|
open("/home/masa/work/test.dat","a"){|f| f.print "key=", key, " value=", value, "\n"}
...

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

masa@masa ~/work $ cat test.dat 
getin process
@request_path=/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat
getin import_user_input
key=language value=de
key=invoice value=25518097
key=event value=download
key=default_flavor value=gcc
key=filename value=test.dat
key=flavor value=gcc
key=email value=aaa@bbb.ccc

Notes

• import_user_input method analyze the URL string
• The data is divided as hash data
• URL is divided into /(language)/(flavor)/(event)/(key)/(value)/(key)/(value)/(key)/(value)
• This process runs even if there is no downloaded file

Consideration

• Probably this import_user_input method mainly decides the state of session and calls the appropriate HTML code by HtmlGrid
• But this case is not to call a html code, but just to download a file, in other words, to delegate it to the actual path to a file
• Maybe, the detail process after the URL string analysis is delegated to the state module in oddb.org

Experiment

src/state/global.rb#download

            def download
open("/home/masa/work/test.dat","a"){|f| f.print "getin src/state/global.rb#download\n"}
open("/home/masa/work/test.dat","a"){|f| f.print caller(0).pretty_inspect, "\n"}
...

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

masa@masa ~/work $ cat test.dat 
getin process
@request_path=/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat
getin import_user_input
key=language    value=de
key=invoice     value=25518097
key=event       value=download
key=default_flavor      value=gcc
key=filename    value=test.dat
key=flavor      value=gcc
key=email       value=aaa@bbb.ccc
getin import_cookies
key=language    value=de
getin src/state/global.rb#download
["/home/masa/ywesee/oddb.org/src/state/global.rb:221:in `download'",
 "/usr/lib64/ruby/1.8/open-uri.rb:32:in `open_uri_original_open'",
 "/usr/lib64/ruby/1.8/open-uri.rb:32:in `open'",
 "/home/masa/ywesee/oddb.org/src/state/global.rb:221:in `download'",
 "/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:214:in `send'",
 "/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:214:in `_trigger'",
 "/home/masa/ywesee/oddb.org/src/state/global.rb:657:in `_trigger'",
 "/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:203:in `trigger'",
 "/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:412:in `process'",
 "/home/masa/ywesee/oddb.org/src/util/session.rb:111:in `process'",
 "/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:176:in `drb_process'",
 "/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:174:in `synchronize'",
 "/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:174:in `drb_process'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `__send__'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `perform_without_block'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1516:in `perform'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1590:in `main_loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `main_loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `start'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `main_loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1431:in `run'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `start'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `run'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1348:in `initialize'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `new'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `start_service'",
 "bin/oddbd:38"]

Experiment

/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb#drb_process

    def drb_process(request)
open("/home/masa/work/test.dat","a"){|f| f.print "SBSM::Session#getin-drb_process\n"}
      start = Time.now
      html = @mutex.synchronize do
        process(request)
        to_html
      end
      (@@stats[@request_path] ||= []).push(Time.now - start)
return_value = html
open("/home/masa/work/test.dat","a"){|f| f.print "html=", return_value, "\n"}
#      html
return_value
    end

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

masa@masa ~/work $ cat test.dat 
SBSM::Session#getin-drb_process
html=

Note

• There is no html code returned from SBSM::Settion#drb_process method

Consideration

• There must be a way used to indicate the target download file from the session information
• It is certain that the html return value of the drb_process method is not used for the downloading

Experiment

/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb#process

        def process(request)
            begin
                @request = request
        @request_method = request.request_method
        @request_path = @request.unparsed_uri
                @validator.reset_errors() if @validator
                import_user_input(request)
                import_cookies(request)
                @state = active_state.trigger(event())
        #FIXME: is there a better way to distinguish returning states?
        #       ... we could simply refuse to init if event == :sort, but that
        #       would not solve the problem cleanly, I think.
#        unless(@state.request_path)
#          @state.request_path = @request_path
#          @state.init
#        end
#               unless @state.volatile?
#                   @active_state = @state
#                   @attended_states.store(@state.object_id, @state)
#               end
#               @zone = @active_state.zone
#               @active_state.touch
#               cap_max_states

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

• Download succeeds

Experiment /usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb#process

        def process(request)
            begin
                @request = request
        @request_method = request.request_method
        @request_path = @request.unparsed_uri
                @validator.reset_errors() if @validator
                import_user_input(request)
                import_cookies(request)


#                @state = active_state.trigger(event())


        #FIXME: is there a better way to distinguish returning states?
        #       ... we could simply refuse to init if event == :sort, but that
        #       would not solve the problem cleanly, I think.
#        unless(@state.request_path)
#          @state.request_path = @request_path
#          @state.init
#        end
#               unless @state.volatile?
#                   @active_state = @state
#                   @attended_states.store(@state.object_id, @state)
#               end
#               @zone = @active_state.zone
#               @active_state.touch
#               cap_max_states

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

• Download fails

Notes

• @state = active_state.trigger(event()) is the key statement
• event() = 'download'
• 'active_state.trigger(event())' decides the State class
• trigger method is defined in sbsm/lib/sbsm/state.rb
• It seems that there are several patterns to choose a State class
• The return value of 'event()' is the 3rd value of URL string, z.B., /abc/def/download/123/456 then 'download' becomes the return value of 'event()'

Experiment /usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb#process

        def process(request)
            begin
                @request = request
        @request_method = request.request_method
        @request_path = @request.unparsed_uri
                @validator.reset_errors() if @validator
                import_user_input(request)
                import_cookies(request)


                @state = active_state.trigger(event())
open("/home/masa/work/test.dat","a"){|f| f.print "@state.class=", @state.class, "\n"}

        #FIXME: is there a better way to distinguish returning states?
        #       ... we could simply refuse to init if event == :sort, but that
        #       would not solve the problem cleanly, I think.
#        unless(@state.request_path)
#          @state.request_path = @request_path
#          @state.init
#        end
#               unless @state.volatile?
#                   @active_state = @state
#                   @attended_states.store(@state.object_id, @state)
#               end
#               @zone = @active_state.zone
#               @active_state.touch
#               cap_max_states

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

@state.class=ODDB::State::User::Download

Experiment

src/view/user/download.rb

module ODDB
    module View
        module User
class Download < HtmlGrid::PassThru
    def init
open("/home/masa/work/test.dat","a"){|f| f.print "getin src/view/user/download.rb#init\n"}
        if(filename = @session.user_input(:filename))
            @path = File.join('..', 'data', 'downloads', filename)
        end
    end
    def to_html(context)
open("/home/masa/work/test.dat","a"){|f| f.print "getin src/view/user/download.rb#to_html\n"}
        line = [
            nil,
            @session.remote_addr,
            @session.user_input(:email),
            @path,
        ].join(';')
        LogFile.append(:download, line, Time.now)
        @session.passthru(@path)
        ''
    end
end
        end
    end
end

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

@state.class=ODDB::State::User::Download
getin src/view/user/download.rb#init
getin src/view/user/download.rb#to_html

Notes

• ODDB::View::User::Download (SBSM::View) class is called from ODDB::State::User::Download (SBSM::State) class, actually src/state/global.rb#download method
• In the process method of SBSM::Session, when the @state is decided, the corresponding View class looks automatically decided
• and the corresponding logic (html code, HtmlGrid) looks chosen.

Experiment

src/state/global.rb#download

            def download
open("/home/masa/work/test.dat","a"){|f| f.print "getin src/state/global.rb#download\n"}
                if(@session.is_crawler?)
                    return State::Drugs::Init.new(@session, nil)
                end
                email = @session.user_input(:email)
                email ||= @session.get_cookie_input(:email)
                oid = @session.user_input(:invoice)
                file = @session.user_input(:filename)
return_value = nil
                if((invoice = @session.invoice(oid)) \
          && invoice.yus_name == email \
                    && invoice.payment_received? \
                    && (item = invoice.item_by_text(file)) \
                    && !item.expired?)
                    #State::User::Download.new(@session, item)
return_value = State::User::Download.new(@session, item)
                else
                    #State::PayPal::Return.new(@session, invoice)
return_value = State::PayPal::Return.new(@session, invoice)
                end
open("/home/masa/work/test.dat","a"){|f| f.print "return_value=", return_value, "\n"}
return return_value

           end

Reboot oddb.org/bin/oddbd

Access to 'http://oddb.masa.org/de/gcc/download/invoice/25518097/email/aaa@bbb.ccc/filename/test.dat'

Result

masa@masa ~/work $ cat test.dat 
getin src/state/global.rb#download
return_value=#<ODDB::State::User::Download:0x7f54fa0b1c78>

Access to 'http://oddb.masa.org/de/gcc/download/invoice/12345/email/aaa@bbb.ccc/filename/test.dat'

Result

getin src/state/global.rb#download
return_value=#<ODDB::State::PayPal::Return:0x7f54fb4f72f0>

Consideration

• Namely, invoice = @session.invoice(oid) is used in order to check the expiration of the user

Just for information

• The access log is saved in log directory

masa@masa ~/ywesee/oddb.org $ cat log/download/2010/11.log 
2010-11-15 15:05:49 CET;127.0.0.1;aaa@bbb.ccc;../data/downloads/test.dat

Core part

src/view/user/download.rb

module ODDB
    module View
        module User
class Download < HtmlGrid::PassThru
    def init
#open("/home/masa/work/test.dat","a"){|f| f.print "getin src/view/user/download.rb#init\n"}
        if(filename = @session.user_input(:filename))
            @path = File.join('..', 'data', 'downloads', filename)
        end
    end

    def to_html(context)
#open("/home/masa/work/test.dat","a"){|f| f.print "getin src/view/user/download.rb#to_html\n"}
#open("/home/masa/work/test.dat","a"){|f| f.print caller(0).pretty_inspect}
        line = [
            nil,
            @session.remote_addr,
            @session.user_input(:email),
            @path,
        ].join(';')
        LogFile.append(:download, line, Time.now)
        @session.passthru(@path)
        ''
    end

Notes

• 'init' method sets the actual path to the download file to @path variable
• In the 'to_html' method, @session.passthru(@path) means (probably) that we can directly access to the file by the browser

General structure (flow) of SBSM and HtmlGrid

• A View class inherits HtmlGrid class which defines usually the corresponding html source code,
• and the view class is called automatically from a State class which inherits SBSM::State class
• A State class is chosen in the 'process' method of a Session class which inherits SBSM::Session class

Check de.oddb.org structure (View and State classes)

grep search

masa@masa ~/ywesee/de.oddb.org $ grep -r passthru lib
lib/oddb/html/view/download.rb:require 'htmlgrid/passthru'
lib/oddb/html/view/download.rb:    @session.passthru(@model)

lib/oddb/html/view/download.rb

require 'htmlgrid/passthru'

module ODDB
  module Html
    module View
class Download < HtmlGrid::PassThru
  def to_html(context)
    @session.passthru(@model)
    ''
  end
end
    end
  end
end

lib/oddb/html/state/download.rb

require 'oddb/html/state/global_predefine'
require 'oddb/html/view/download'

module ODDB
  module Html
    module State
class Download < State::Global
    VIEW = View::Download
    VOLATILE = true
end
    end
    end
end

lib/oddb/html/state/global.rb

  def _download(file)
    path = File.join ODDB.config.export_dir, file
    if File.exist?(path)
      Download.new(@session, path)
    elsif match = /(.+)_(.+).csv/.match(file)
      packages = _search_local match[1].tr('-', ' '), match[2]
      packages.filename = file
      Drugs::DownloadExport.new(@session, packages)
    end
  end

lib/oddb/html/util/session.rb

  def process(request)
    @request_path = request.unparsed_uri
    @process_start = Time.now
    super
    if(!is_crawler? && lookandfeel.enabled?(:query_limit))
      limit_queries
    end
    '' ## return empty string across the drb-border
  end

Notes

• Basically, the file structure is similar to one of oddb.org
• But it seems that something has been simplified
• In particular, _download method is quite different from download method of oddb.org
• and there is no 'download' method defined in src/state/global.rb
• In de.oddb.org, View and State classes are under lib/html directory

Experiments in de.oddb.org

(Tomorrow)