view · edit · sidebar · attach · print · history

20110919-testcases-export-sort-migel

<< | Index | >>

  1. Testcases
  2. Improve importer
  3. Update sort function as well as chde.oddb.org
  4. Switch 'call by reference' to 'call by value'

Goal/Estimate/Evaluation:

  • Migel server / 90% / 90%
Milestones
  1. Testccase
  2. Import
  3. Sort as well as de and chde
  4. Testcase (report process)
  5. Importer job
ToDo
  • require 'migel/util/m10l_document' problem in multilingual.rb
  • require 'oddb/util/m10l_document' problem in model_super.rb

Testcases

Improve importer

Task

  • report
  • reimport missing data

New file

Updated files

  • lib/migel/config.rb 
module Migel
...
  defaults = {
...
    'admins'            => [],
    'mail_from'         => 'update@ywesee.com',
    'mail_charset'      => 'utf8',
    'smtp_authtype'     => :plain,
    'smtp_domain'       => 'ywesee.com',
    'smtp_pass'         => nil,
    'smtp_port'         => 587,
    'smtp_server'       => 'localhost',
    'smtp_user'         => 'update@ywesee.com',
  }
  • etc/migel.yml 
---
db_user: 'postgres'
db_auth: ''

admins:
 - mhatakeyama@ywesee.com
mail_from: mhatakeyama@ywesee.com
smtp_domain: ywesee.com
smtp_server: smtp.gmail.com
smtp_user: mhatakeyama@ywesee.com
smtp_pass: 'xxxx'

Run

  • bin/migeld
  • bin/admin 
migel> Migel::Util::Mail.notify_admins('test', ['hogehoge'])

Result 

From	mhatakeyama@ywesee.com
To	mhatakeyama@ywesee.com

hogehoge

Experiment

  • lib/migel/util/importer.rb 
  def reported_save_all_products
    lines = [
      sprintf("%s: %s %s#save_all_products", Time.now.strftime('%c'), Migel.config.server_name, self.class)
    ]
    #save_all_products
    lines.concat report
  rescue Exception => err
    lines.push(err.class.to_s, err.message, *err.backtrace)
    lines.concat report
  ensure
    subject = lines[0]
    Mail.notify_admins(subject, lines)
  end
  def report
    ['report']
  end

Run

  • bin/admin 
migel> Migel::Util::Importer.new.reported_save_all_products
-> Mon Sep 19 09:49:42 2011: migel Migel::Util::Importer#save_all_productsreport

Result

Experiment

  • lib/migel/importer.rb 
  def report
    [
      sprintf("Saved file: #{@saved_csv_file}"),
      sprintf("Total %5i Migelids (Saved %5i / Unsaved %5i):", migel_code_list.length, @saved_migelids, @unsaved_migelids),
      sprintf("Saved %5i Products:", @saved_products),
      sprintf("Save time length: #{@save_time_length}"),
    ]
  end

  def save_all_products(file_name = 'migel_products_de.csv', lang = 'de', estimate = false)
    @saved_products = 0
    @saved_migelids = 0
    @unsaved_migelids = 0
...
    CSV.open(file_name, 'w') do |writer|
      migel_codes.each_with_index do |migel_code, count|
...

          @saved_products += 1
          product_flag = true
        end
        if product_flag
          @saved_migelids += 1
        else
          @unsaved_migelids += 1
        end
        time = estimate_time(start_time, total, count+1)
        @save_time_length = time.split(':')[1].split(' ')[0,2].join(' ')
        puts time if estimate
      end # migel_codes
    end # CSV
  end

Run

  • bin/admin 
migel> Migel::Util::Importer.new.reported_save_all_products('hogehoge.csv', 'de', true)

Result

  • Email

Experiment

  • bin/admin 
migel> Migel::Util::Importer.new.missing_article_name_migel_code_list('de', 'missing_migel_code_list.dat')

masa@masa ~/ywesee/migel_dev $ cat missing_migel_code_list.dat
01.01.01.00.1
01.01.02.00.2
03.03.01.00.1
...
masa@masa ~/ywesee/migel_dev $ head migel_product_de.csv 
01.01.01.00.1,1624501,7612479001117,AMEDA Einhandmilchpumpe mit Flexishield,Nufer AG (Medizintechnik),,32.30,43.70,1,43.70,,A,2010-08-16T00:00:00+00:00,2010-07-09T00:00:00+00:00,DE
01.01.01.00.1,4637852,,AMEDA LACT-H Zweihandpumpe,Nufer AG,,17.80,24.10,1,24.10,,,,,
01.01.01.00.1,4443763,7640109051744,ARDO AMARYLL Individuelle Handmilchpumpe,Ardo medical AG,,41.30,74.00,1,74.00,,A,2010-08-16T00:00:00+00:00,2010-03-15T00:00:00+00:00,DE
01.01.01.00.1,1930445,8710103528920,AVENT PHILIPS ISIS Milchpumpe manuell,Philips AG (Haushalt),,60.30,95.00,1,95.00,100,A,2011-02-02T00:00:00+00:00,1997-12-22T00:00:00+00:00,DE
...

Note

  • The missing data is save in the csv file. But the article names of those products are not shown in the view.
  • It may be not a problem of online migel server but a problem of saving from a csv file

Note

  • Import products data from a csv again

Result

Note

  • There are not same
  • There is no fr product data missing
  • If the 'Migel::Util::Importer.new.import_all_products_from_csv('migel_product_de.csv', 'de', true)' runs twice, then there is no missing datat
  • There are 2 possibilities to fail in the importing
    1. when creating an instance of product
    2. when saving an instance of product

suspend

Update sort function as in chde.oddb.org

Task

  • update sort function in migel as in chde.oddb.org
  • chde.oddb.org can show the data in both ch.oddb.org and chde.oddb.org at the same time and it can sort the data

de.oddb.org

sort function

  • lib/oddb/html/util/sort.rb

sort link

  • lib/oddb/html/view/list.rb

create a remote object

  • lib/oddb/html/state/drugs/global.rb
  • (lib/oddb/export/xls.rb)

Hypothesis

  • Probably a remote DRbObject is created in the side of de.oddb.org 'calling by value', not 'calling by reference'
  • That is why Remote objects are defined in the side of de.oddb.org

Experiment

  • person.rb 
require 'drb'

class Person
  include DRbUndumped
  attr_accessor :n
  def initialize(n)
    @n = n
  end
end
  • server.rb 
require 'drb'
require 'person'

here = 'druby://localhost:17640'
@queue = Queue.new
arr = []
10.times {|i|
  @queue.push(Person.new(i))
}

DRb.start_service(here, @queue)
  • client.rb 
require 'drb'
#require 'person'
there = 'druby://localhost:17640'
DRb.start_service(nil)
@queue = DRbObject.new(nil, there)

Run

  • terminal 1 
masa@masa ~/work/drb_error_test $ irb
irb(main):001:0> require 'server'
=> true
irb(main):002:0> @queue.length
=> 10
  • terminal 2 
masa@masa ~/work/drb_error_test $ irb
irb(main):001:0> require 'client'
=> true
irb(main):002:0> @queue.length
=> 10
irb(main):003:0> q = @queue.pop
=> #<DRb::DRbObject:0x7fe306620200 @uri="druby://localhost:17640", @ref=69898410693460>
irb(main):004:0> q.n
=> 0
irb(main):005:0> @queue.length
=> 9
  • terminal 1 
irb(main):004:0> GC.start
=> nil
  • terminal 2 
irb(main):006:0> q.n
RangeError: 0x3fbddfdae01c is recycled object

Note

  • If it is NOT possible to do Marshal.dump an object , the object is 'called by reference' via druby
  • The Module DRb::Undumped make the object an unmarshaled object
  • One of the merits of 'call by referece' is to be able to call methods of the server side object from the client without class definition
  • The object in the client side becomes 'DRbObject'

Experiment

  • person.rb require 'drb'
    class Person 
    #include DRbUndumped
    attr_accessor :n
    def initialize(n)
      @n = n
    end
end
  • client.rb require 'drb' #require 'person' there = 'druby://localhost:17640' DRb.start_service(nil) @queue = DRbObject.new(nil, there)

Run

  • terminal 1 
masa@masa ~/work/drb_error_test $ irb
irb(main):001:0> require 'server'
=> true
irb(main):002:0> @queue.length
=> 10
  • terminal 2 
masa@masa ~/work/drb_error_test $ irb
irb(main):001:0> require 'client'
=> true
irb(main):002:0> @queue.length
=> 10
irb(main):003:0> q = @queue.pop
=> #<DRb::DRbUnknown:0x7fe37e4b1c98 @buf="\004\bo:\vPerson\006:\a@ni\000", @name="Person">
irb(main):004:0> q.n
NoMethodError: undefined method `n' for #<DRb::DRbUnknown:0x7fe37e4b1c98>
        from (irb):4

Note

  • If it is possible to do Marshal.dump an object, the object is 'called by value' in the client side
  • It is impossible to call methods without class definition in the object called by value in the client side

Experiment

  • client.rb require 'drb' require 'person' there = 'druby://localhost:17640' DRb.start_service(nil) @queue = DRbObject.new(nil, there)

Run

  • terminal 1 
masa@masa ~/work/drb_error_test $ irb
irb(main):001:0> require 'server'
=> true
irb(main):002:0> @queue.length
=> 10
  • terminal 2 
masa@masa ~/work/drb_error_test $ irb
irb(main):001:0> require 'client'
=> true
irb(main):002:0> @queue.length
=> 10
irb(main):003:0> q = @queue.pop
=> #<Person:0x7f1c49df9cf8 @n=0>
irb(main):004:0> q.n
=> 0
irb(main):005:0> @queue.length
=> 9
  • terminal 1 
irb(main):003:0> @queue.length
=> 9
irb(main):004:0> GC.start
=> nil
  • terminal 2 
irb(main):006:0> q.n
=> 0

Note

  • The object called by value is a COPY of the server side object
  • If the object of the server is deleted the client object called by value is left independently from the server side object
  • but the client side needs the class definition in order to call a method of the object

Reference

Switch 'call by reference' to 'call by value'

view · edit · sidebar · attach · print · history
Page last modified on September 26, 2011, at 08:30 AM