view · edit · sidebar · attach · print · history

20101112-debug-export_chde_xls analyze-grant_download-command

<< | Index | >>

  1. Confirm the error of export_chde_xls
  2. Run jobs/export_chde_xls manually on production server
  3. Check bin/admin de.oddb.org
  4. Check source code of bin/admin
  5. Define grant_download method
Goal
  • Debug export_chde_xls / 100%
Milestones
  1. Confirm the error
  2. Run jobs/export_chde_xls on production server manually success
  3. Confirm bin/admin in both oddb.org and de.oddb.org 11:45
  4. Check source code of bin/admin 15:00 Found the cause
  5. Define grant_download method
Summary
Commits
ToDo Tomorrow
  • Define grant_download method in de.oddb.org
Keep in Mind
Attached Files

Confirm the error of export_chde_xls

Error email Mon Nov 1 04:09:08 2010: ODDB::Export::Xls::ComparisonDeCh 

Mon Nov  1 04:09:08 2010: ODDB::Export::Xls::ComparisonDeCh#export
NoMethodError
undefined method `active_agents' for nil:NilClass
(druby://localhost:10013) /usr/lib64/ruby/site_ruby/1.8/odba/stub.rb:112:in `send'
(druby://localhost:10013) /usr/lib64/ruby/site_ruby/1.8/odba/stub.rb:112:in `method_missing'
(druby://localhost:10013) /var/www/oddb.org/src/model/part.rb:31:in `active_agents'
(druby://localhost:10013) /usr/lib64/ruby/site_ruby/1.8/odba/stub.rb:112:in `send'
(druby://localhost:10013) /usr/lib64/ruby/site_ruby/1.8/odba/stub.rb:112:in `method_missing'
(druby://localhost:10013) /var/www/oddb.org/src/model/package.rb:68:in `active_agents'
(druby://localhost:10013) /usr/lib64/ruby/1.8/drb/drb.rb:395:in `inject'
(druby://localhost:10013) /var/www/oddb.org/src/model/package.rb:68:in `each'
(druby://localhost:10013) /var/www/oddb.org/src/model/package.rb:68:in `inject'
(druby://localhost:10013) /var/www/oddb.org/src/model/package.rb:68:in `active_agents'
(druby://localhost:11001) /var/www/de.oddb.org/lib/oddb/remote/drugs/package.rb:27:in `active_agents'
(druby://localhost:11001) /var/www/de.oddb.org/lib/oddb/remote/drugs/package.rb:75:in `local_comparables'
(druby://localhost:11001) /var/www/de.oddb.org/lib/oddb/export/xls.rb:27:in `collect_comparables'
(druby://localhost:11001) /usr/lib64/ruby/1.8/drb/drb.rb:1555:in `call'
(druby://localhost:10013) /usr/lib64/ruby/1.8/drb/invokemethod.rb:10:in `block_yield'
(druby://localhost:10013) /usr/lib64/ruby/1.8/drb/invokemethod.rb:17:in `perform_with_block'
(druby://localhost:10013) /var/www/oddb.org/ext/readonly/src/readonly_server.rb:40:in `call'
(druby://localhost:10013) /var/www/oddb.org/ext/readonly/src/readonly_server.rb:40:in `remote_each_package'
(druby://localhost:10013) /var/www/oddb.org/ext/readonly/src/readonly_server.rb:38:in `each'
(druby://localhost:10013) /var/www/oddb.org/ext/readonly/src/readonly_server.rb:38:in `remote_each_package'
/var/www/de.oddb.org/lib/oddb/export/xls.rb:22:in `collect_comparables'
/var/www/de.oddb.org/lib/oddb/export/xls.rb:15:in `export'
/var/www/de.oddb.org/lib/oddb/export/server.rb:57:in `safe_export'
/usr/lib64/ruby/1.8/tempfile.rb:172:in `open'
/var/www/de.oddb.org/lib/oddb/export/server.rb:54:in `safe_export'
/var/www/de.oddb.org/lib/oddb/export/server.rb:25:in `export_chde_xls'
/var/www/de.oddb.org/lib/oddb/export/server.rb:39:in `run'
/var/www/de.oddb.org/lib/oddb/export/server.rb:48:in `call'
/var/www/de.oddb.org/lib/oddb/export/server.rb:48:in `on_monthday'
/var/www/de.oddb.org/lib/oddb/export/server.rb:38:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1515:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1589:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1430:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1347:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `start_service'
/var/www/de.oddb.org/bin/exportd:35

Notes

  • druby 10013 = oddb.org/ext/readonly/bin/readonlyd
  • druby 11001 = ODDB::Exporter thread (it is called from de.oddb.org)

Reviews

Test locally

Set up

  • run de.oddb.org (de.oddb.org/bin/oddbd)
  • run currencyd (currency/bin/currencyd)
  • run readonlyd (oddb.org/ext/readonly/bin/readonlyd)
  • run jobs/export_chde_xls (de.oddb.org/jobs/export_chde_xls)

In order to run readonlyd, I have to change the require path for multilingual 

from

  oddb.org/src/remote/galenic_group.rb:#require 'oddb/util/multilingual'
  oddb.org/src/remote/galenic_form.rb:#require 'oddb/util/multilingual'

to

  oddb.org/src/remote/galenic_group.rb:require 'src/remote/multilingual'
  oddb.org/src/remote/galenic_form.rb:require 'src/remote/multilingual'

Refer

  • de.oddb.org README

Result

  • There is no error mail
  • The files are created in var/downloads directory 
masa@masa ~/ywesee/de.oddb.org/var/downloads $ ls -al
insgesamt 1448
drwxr-xr-x 2 masa masa     72 12. Nov 09:21 .
drwxr-xr-x 3 masa masa      8 12. Nov 08:53 ..
-rw-r--r-- 1 masa masa 962048 12. Nov 09:12 chde.xls
-rw-r--r-- 1 masa masa 255520 12. Nov 09:12 chde.xls.gz
-rw-r--r-- 1 masa masa 255618 12. Nov 09:12 chde.xls.zip

Run jobs/export_chde_xls manually on production server

Confirmation files on production server 

de.oddb.org/var/downloads $ ls -al
total 1297268
drwxrwxr-x  2 apache apache      4096 2010-11-12 02:37 .
drwxr-xr-x 11 apache apache        80 2010-03-05 18:05 ..
-rw-r--r--  1 ywesee users     955904 2010-10-22 09:37 chde.xls
-rw-------  1 apache apache         0 2010-10-01 04:00 chde.xls.666.0
-rw-------  1 apache apache         0 2010-11-01 04:00 chde.xls.666.1
-rw-r--r--  1 ywesee users     253784 2010-10-22 09:37 chde.xls.gz
-rw-r--r--  1 ywesee users     253880 2010-10-22 09:37 chde.xls.zip

Notes

  • The new file is certainly not created

Run de.oddb.org/jobs/export_chde_xls manually on production server

Result 

/var/www/de.oddb.org $ ls var/downloads/ -al
total 1297276
drwxrwxr-x  2 apache apache      4096 2010-11-12 10:09 .
drwxr-xr-x 11 apache apache        80 2010-03-05 18:05 ..
-rw-r--r--  1 ywesee users     966144 2010-11-12 10:08 chde.xls
-rw-r--r--  1 ywesee users     257057 2010-11-12 10:09 chde.xls.gz
-rw-r--r--  1 ywesee users     257154 2010-11-12 10:09 chde.xls.zip

Check bin/admin de.oddb.org

Email from Davatz-san

Abstract

  • There is a command in bin/admin of ch.oddb.org for setting the download for a certain user for certain data. 
 grant_download 'email@domain.com', 'oddb.csv', 17, Time.local(2011,2,11) - month, day

Confirmation

on oddb.org (ch.oddb.org)

grant_download test 

masa@masa ~/ywesee/oddb.org $ bin/admin 
ch.oddb> grant_download 'mhatakeyama@ywesee.com', 'oddb.csv', 11, Time.local(2011,11,12)
-> http://oddb.masa.org/de/gcc/download/invoice/25503676/email/mhatakeyama@ywesee.com/filename/oddb.csv

Notes

  • The command looks accepted

on de.oddb.org

grant_download test 

masa@masa ~/ywesee/de.oddb.org $ bin/admin 
de.oddb> grant_download 'mhatakeyama@ywesee.com', 'chde.xls', 11, Time.local(2011,11,12)
-> undefined method `grant_download' for nil:ODDB::Util::Server

oddbd log 

E, [2010-11-12T13:32:42.619649 #8086] ERROR -- admin: NoMethodError
E, [2010-11-12T13:32:42.619717 #8086] ERROR -- admin: undefined method `grant_download' for nil:ODDB::Util::Server
E, [2010-11-12T13:32:42.620289 #8086] ERROR -- admin: ["/home/masa/ywesee/de.oddb.org/lib/oddb/util/server.rb:31:in `_admin'",
 "(eval):1:in `_admin'",
 "/home/masa/ywesee/de.oddb.org/lib/oddb/util/server.rb:28:in `initialize'",
 "/home/masa/ywesee/de.oddb.org/lib/oddb/util/server.rb:28:in `new'",
 "/home/masa/ywesee/de.oddb.org/lib/oddb/util/server.rb:28:in `_admin'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `__send__'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1556:in `perform_without_block'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1516:in `perform'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1590:in `main_loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1586:in `main_loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `start'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1582:in `main_loop'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1431:in `run'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `start'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1428:in `run'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1348:in `initialize'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `new'",
 "/usr/lib64/ruby/1.8/drb/drb.rb:1628:in `start_service'",
 "bin/oddbd:24"]

Notes

  • The command looks NOT defined

Check source code of bin/admin

oddb.org (ch.oddb.org)

grep search 

masa@masa ~/ywesee/oddb.org $ grep -r grant_download *
src/util/oddbapp.rb:    def grant_download(email, filename, price, expires=Time.now+2592000)

bin/admin 

#!/usr/bin/env ruby
# Admin -- oddb -- 09.07.2003 -- hwyss@ywesee.com 

$0 = "ch.oddb (Admin-Tool)"

$: << File.expand_path('../src', File.dirname(__FILE__))

require 'rubygems'
require 'drb'
require 'util/oddbconfig'
require 'readline'
include Readline

# There is no definition of RequestStub in de.oddb.org
class RequestStub
    def ask(meth, *args)
        pr = "call: #{meth}(#{args.join(',')}) return? > "
        readline(pr, true)
    end
    def cookies
        instance_eval(ask(:cookies)) || {}
    end
    def params
        instance_eval(ask(:params)) || {}
    end
    def unparsed_uri
        ask(:unparsed_uri)
    end
    def user_agent
        ask(:user_agent)
    end
end

# This part is totally the same as de.oddb.org
Kernel.trap('INT') {
    if($current.respond_to?(:exit))
        $current.exit
        $current = nil
    else
        puts "<ctrl-c> is used to kill admin-threads."
        puts "Type <quit> or <exit> to exit."
    end
}

prompt = 'ch.oddb> '
req = RequestStub.new                     # This is only in oddb.org
request = DRb.start_service('druby://localhost:0', req)
oddb = DRbObject.new(nil, ODDB::SERVER_URI)
last_error = nil
priority = 0
while(line = readline(prompt, true))
    result = ''
    result.extend(DRb::DRbUndumped)
    line.strip!
    $stdout.print "-> "
    case line.downcase
    when 'exit', 'quit'
        $stdout.puts 'Goodbye'
        break
    when 'bt', 'backtrace'
        if last_error.respond_to?(:backtrace)
            $stdout.puts last_error.backtrace
        else
            $stdout.puts 'No Error'
        end
    when /^last_error/
        begin
            $stdout.puts eval(line)
        rescue StandardError, SyntaxError => last_error
            $stdout.puts last_error
        end

# Until here, it is exactly the same as de.oddb.org, but
# After here it is different from de.oddb.org
    when 'process'
        cmd = "self['test'].process(DRbObject.new(nil, '#{request.uri}'))"
        begin
            $current = oddb._admin(cmd, result, priority)
            $current.join
            $stdout.puts result
        rescue StandardError, SyntaxError => last_error
            $stdout.puts last_error
        end
    when /to_html(\[[\d,.]+\])?/
        begin
            range = $1 || "[0,200]"
            cmd = "self['test'].to_html#{range}"
            $current = oddb._admin(cmd, result, priority)
            $current.join
            $stdout.puts result
        rescue StandardError, SyntaxError => last_error
            $stdout.puts last_error
        end
# After here, the same as de.oddb.org
    when /sp\s+(-?\d+)/, /set_priority\s+(-?\d+)/, /^sp$/
        priority = $~[1].to_i
        $stdout.puts "setting Priority to #{priority}"
    else
        begin
            bgflag = line.slice!(/\s*\&\s*$/)
            $current = oddb._admin(line, result, priority)
            unless(bgflag)
                $current.join
            end
            $stdout.puts result
        rescue StandardError, SyntaxError => last_error
            $stdout.puts last_error
        end
    end
    $current = nil
    $stdout.flush
end

Notes

  • The differences between oddb.org and de.oddb.org are mainly
    • the definition of class RequestStub
    • when conditions, 'process', /to_html(\[[\d,.]+\])?/

de.oddb.org

grep search 

masa@masa ~/ywesee/de.oddb.org $ grep -r grant_download *
masa@masa ~/ywesee/de.oddb.org $

bin/admin 

#!/usr/bin/env ruby
# Admin -- oddb -- 09.07.2003 -- hwyss@ywesee.com 

$0 = "de.oddb (Admin-Tool)"

$:.push(File.expand_path('../lib', File.dirname(__FILE__)))

$KCODE = 'u'

require 'drb'
require 'sbsm/drb'
require 'oddb'
require 'oddb/config'
require 'encoding/character/utf-8'
require 'readline'
include Readline

Kernel.trap('INT') {
  if($current.respond_to?(:exit))
    $current.exit
    $current = nil
  else
    puts "<ctrl-c> is used to kill admin-threads."
    puts "Type <quit> or <exit> to exit."
  end
}

prompt = 'de.oddb> '
DRb.start_service('druby://localhost:0')
oddb = DRbObject.new(nil, ODDB.config.server_url)
last_error = nil
priority = -1
while(line = readline(prompt, true))
  result = ''
  result.extend(DRb::DRbUndumped)
  line.strip!
  $stdout.print "-> "
  case line.downcase
  when 'exit', 'quit'
    $stdout.puts 'Goodbye'
    break
  when 'bt', 'backtrace'
    if last_error.respond_to?(:backtrace)
      $stdout.puts last_error.backtrace
    else
      $stdout.puts 'No Error'
    end
  when /^last_error/
    begin
      $stdout.puts eval(line)
    rescue StandardError, SyntaxError => last_error
      $stdout.puts last_error
    end
  when /sp\s+(-?\d+)/, /set_priority\s+(-?\d+)/
    priority = $~[1].to_i
    $stdout.puts "setting Priority to #{priority}"
  else
    begin
      bgflag = line.slice!(/\s*\&\s*$/)
      $current = oddb._admin(line, result, priority)
      unless(bgflag)
        last_error = $current.join
      end
      $stdout.puts result
    rescue StandardError, SyntaxError => last_error
      $stdout.puts last_error
    end
  end
  $current = nil
  $stdout.flush
end

Notes

  • There is not class RequestStub definition
  • There is not when condition, 'process' and /to_html(\[[\d,.]+\])?/

Understand the code

Notes

  • Generally, 'line.downcase' becomes command string
  • Each command runs depending on the when case
  • For example, when we type 'exit', then 
        $stdout.puts 'Goodbye'
        break

run.

If we type 'grant_download ...', the 'else' part 

    else
        begin
            bgflag = line.slice!(/\s*\&\s*$/)
            $current = oddb._admin(line, result, priority)
            unless(bgflag)
                $current.join
            end
            $stdout.puts result
        rescue StandardError, SyntaxError => last_error
            $stdout.puts last_error
        end
    end

run in both oddb.org and de.oddb.org.

  • Main part is 'oddb._admin(line, result, priority)'
  • oddb =
    • DRbObject.new(nil, ODDB::SERVER_URI) (oddb.org)
    • DRbObject.new(nil, ODDB.config.server_url)
  • In either case, oddb is a DRbObject to indicate the server
  • The rest important is '_admin' method

Question

  • How does the methods in src/util/oddbapp.rb are called from here in oddb.org?

grep search _admin (oddb.org) 

masa@masa ~/ywesee/oddb.org $ grep -r "def _admin" *
src/util/oddbapp.rb:            def _admin(src, result, priority=0)

grep search _admin (de.oddb.org) 

masa@masa ~/ywesee/de.oddb.org $ grep -r "def _admin" *
lib/oddb/util/server.rb:      def _admin(src, result, priority=0)

Notes

  • _admin methodi is define in ODDB::Util::Server but the error message says 
-> undefined method `grant_download' for nil:ODDB::Util::Server
  • This 'nil:ODDB::Util::Server' looks strange

Experiment

de.oddb.org/lib/oddb/util/server 

module ODDB
  module Util
    class Server < SBSM::DRbServer
      ENABLE_ADMIN = true
      SESSION = Html::Util::Session
      VALIDATOR = Html::Util::Validator
      def initialize(*args)
        super
        @rss_mutex = Mutex.new
        run_exporter if(ODDB.config.run_exporter)
        run_updater if(ODDB.config.run_updater)
      end
def masa
  "Masa"
end

de.oddb.org/bin/admin 

oddb = DRbObject.new(nil, ODDB.config.server_url)
p oddb.masa

Result 

masa@masa ~/ywesee/de.oddb.org $ bin/admin 
"Masa"
de.oddb>

Notes

  • oddb is certainly ODDB::Util::Server object

Consideration

  • The problem might be in ODDB::Util::Server#_admin method

Experiment 

masa@masa ~/ywesee/de.oddb.org $ bin/admin
de.oddb> masa
-> Masa

Conclusion

  • The error is caused by no definition of 'grant_download' in ODDB::Util::Server class
  • The solution is to define 'grant_download' method in ODDB::Util::Server class as well as oddb.org

Define grant_download method

ToDo

  1. Read grant_download method of oddb.org and understand it
  2. Make a test case
  3. Define the method in lib/oddb/util/server.rb

Check the source code of grant_download method in oddb.org

oddb.org/src/util/oddbapp.rb 

    def grant_download(email, filename, price, expires=Time.now+2592000)
      ip = Persistence::Pointer.new(:invoice)
      inv = update ip.creator, :yus_name => email, :currency => 'EUR'
      itp = inv.pointer + :item
      update itp.creator, :text => filename, :price => price, :time => Time.now,
                          :type => :download, :expiry_time => expires,
                          :duration => (Time.now - expires) / 86400,
                          :vat_rate => 7.6
      inv.payment_received!
      inv.odba_store
      "http://#{SERVER_NAME}/de/gcc/download/invoice/#{inv.oid}/email/#{email}/filename/#{filename}"
    end

Qeustion

  • What does this exactly do?

Notes

  • payment_received! is defined in src/model/invoice.rb
  • Something about invoice is saved in database
  • I guess the url will be valid in one month
  • This command will be executed probably by admin after the order and payment of downloading from a customer
    • If so, the important functions will be as follows:
    • to save, at least, email, expiry_time
    • to output the download url for a certain data file, only for a certain user
  • How is the url translated in oddb.org? => Check SBSM structure
  • Should this method send an email to the email-address or admin user?

grep check 'module Persistence' 

masa@masa ~/ywesee/oddb.org $ grep -r "module Persistence" *
src/util/persistence.rb:        module PersistenceMethods
src/util/persistence.rb:        module Persistence

masa@masa ~/ywesee/de.oddb.org $ grep -r "module Persistence" *
lib/oddb/persistence/odba.rb:  module Persistence
lib/oddb/persistence/og.rb:  module Persistence

grep check 'class Pointer' 

masa@masa ~/ywesee/oddb.org $ grep -r "class Pointer" *
src/util/persistence.rb:                class Pointer
src/util/pointerarray.rb:       class PointerArray < Array
src/util/pointerarray.rb:       class PointerHash < Hash
src/view/pointersteps.rb:               class PointerSteps < HtmlGrid::List
src/view/pointervalue.rb:               class PointerValue < HtmlGrid::Value
src/view/pointervalue.rb:               class PointerLink < View::PointerValue

masa@masa ~/ywesee/de.oddb.org $ grep -r "class Pointer" *
masa@masa ~/ywesee/de.oddb.org $

Notes

  • There is Persistence module in de.oddb.org but not Pointer class
view · edit · sidebar · attach · print · history
Page last modified on December 17, 2010, at 02:04 PM