view · edit · sidebar · attach · print · history

20110920-report-paging-migel

<< | Index | >>

Log reported_save_all_products migel
Check how to use ODBA cache data suspend
Update report mail
Update sort faster
efficient ODBA cache usage migel
Paging function migel

Goal/Estimate/Evaluation:

paging migel / 90% / 90%

Milestones

~~update report~~
1. ~~list migel code with link~~
2. ~~attach a compressed file of result~~
~~update sort (1. company name 2. product)~~
ODBA cache migel
Testcase (report, mail, compress processes)
Importer job

ToDo

require 'migel/util/m10l_document' problem in multilingual.rb
require 'oddb/util/m10l_document' problem in model_super.rb

Log reported_save_all_products migel

Mon Sep 19 11:40:59 2011: migel Migel::Util::Importer#save_all_products

Mon Sep 19 11:40:59 2011: migel Migel::Util::Importer#save_all_products
Saved file: /home/masa/ywesee/migel_dev/data/csv/hogehoge.csv
Total   571 Migelids (Saved   356 / Unsaved   215):
Saved 30742 Products:
Save time length: 7.51 [h]

Note

about 8 hours
5MB file
It may be better to attach the compressed file to the report mail

Check how to use ODBA cache data

Problem

At the moment, it seems that migel server returns search result does not keep in the cache
(the speed of searching is same every time)
Memory cache should be used in migel server
after that, let's think how to sort a search result in the migel server cache
for example, the following search takes the same time length every time
- http://oddb.masa.org/de/gcc/migel_search/migel_code/170103001

Update report mail

Task

List migelids (migel codes) that have products, and do not have products with link
Attach a compressed file

List migel codes

Experiment

lib/migel/util/importer.rb

  def report(lang = 'de')
    lang = lang.downcase
    [
      "Saved file: #{@saved_csv_file}",
      sprintf("Total %5i Migelids (%5i Migelids have products / %5i Migelids have no products)", migel_code_list.length, @migel_codes_with_products.length, @migel_codes_without_products.length),
      "Saved #{@saved_products} Products",
      "Save time length: #{@save_time_length}",
    ].concat([
      '',
      "Migelids with products (#{@migel_codes_with_products.length})"     
    ]).concat(@migel_codes_with_products.sort.map{|migel_code|
      "http://ch.oddb.org/#{lang}/gcc/migel_search/migel_product/#{migel_code}"
    }).concat([
      '',
      "Migelids without products (#{@migel_codes_without_products.length})"     
    ]).concat(@migel_codes_without_products.sort.map{|migel_code|
      "http://ch.oddb.org/#{lang}/gcc/migel_search/migel_product/#{migel_code}"
    })
  end

Run

bin/migeld
bin/admin

migel> Migel::Util::Importer.new.reported_save_all_products('test.csv', 'de', true)
-> Array

Result (Short example)

Email Tue Sep 20 08:38:11 2011: migel Migel::Util::Importer#reported_save_all_products(de)

Tue Sep 20 08:38:11 2011: migel Migel::Util::Importer#reported_save_all_products(de)
Saved file: /home/masa/ywesee/migel_dev/data/csv/test.csv
Total   571 Migelids (    5 Migelids have products /     1 Migelids have no products)
Saved 130 Products
Save time length: 46.13 [m]

Migelids with products (5)
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.01.00.1
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.02.00.2
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.02.02.2
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.02.01.00.2
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.02.01.01.1

Migelids without products (1)
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.02.01.2

Attach a file to report

Reference

ruby gmail http://d.hatena.ne.jp/takuya_1st/20110608/1307505677 (Japanese)
ruby tmail http://d.hatena.ne.jp/mzp/20080304/tmail (Japanese)

Note

de.oddb.org uses RMail class to compose mail body
ch.oddb.org uses both TMail and RMail

Experiment

lib/migel/util/mail.rb

      def Mail.notify_admins_attached(subject, lines, file)
        config = Migel.config
        recipients = config.admins

        # Main part
        mail = TMail::Mail.new
        mail.subject = subject
        mail.date = Time.now
        mail.mime_version = '1.0'

        # Text part
        text = TMail::Mail.new
        text.set_content_type('text', 'plain', 'charset'=>'UTF-8')
        text.body = lines.join("\n")
        mail.parts.push text

        # File part
        attach = TMail::Mail.new
        attach.body = Base64.encode64 File.read(file)
#        attach.set_content_type('image','jpg','name' => file)
        attach.set_content_disposition('attachment', 'filename' => File.basename(file))
        attach.transfer_encoding = 'base64'
        mail.parts.push attach

        sendmail(mail.encoded, config.smtp_user, recipients)
      end

Run

bin/migeld
bin/admin

migel> Migel::Util::Mail.notify_admins_attached('Hoge', ['morning','hello','bye'], '/home/masa/work/hoge.gz')
-> mhatakeyama@ywesee.com

Result

Email Hoge

morning
hello
bye
----
hoge.gz	hoge.gz
1K download

Compress a file

Reference

ruby zlib http://jp.rubyist.net/magazine/?0018-BundledLibraries (Japanese)

test.rb

require 'zlib'
orig = 'hoge'
Zlib::GzipWriter.open('hoge.gz', Zlib::BEST_COMPRESSION) do |gz|
  gz.mtime = File.mtime(orig)
  gz.orig_name = orig
  gz.puts File.open(orig, 'rb'){|f| f.read }
end

Run

masa@masa ~/work $ cat > hoge
hogehoge
foobaa
masa

masa@masa ~/work $ ruby test.rb

Result

masa@masa ~/work $ zcat hoge.gz 
hogehoge
foobaa
masa

Attach a compressed file

Experiment

lib/migel/util/importer.rb

  def reported_save_all_products(file_name = 'migel_products_de.csv', lang = 'de', estimate = false)
    @csv_file = File.join(@data_dir, file_name)
    lines = [
      sprintf("%s: %s %s#reported_save_all_products(#{lang})", Time.now.strftime('%c'), Migel.config.server_name, self.class)
    ]
    save_all_products(@csv_file, lang, estimate)
    compressed_file = compress(@csv_file)
    lines.concat report(lang)
  rescue Exception => err
    lines.push(err.class.to_s, err.message, *err.backtrace)
    lines.concat report
  ensure
    subject = lines[0]
    #Mail.notify_admins(subject, lines)
    Mail.notify_admins_attached(subject, lines, compressed_file)
  end
  def compress(file)
    compressed_filename = file + '.gz'
    Zlib::GzipWriter.open(compressed_filename, Zlib::BEST_COMPRESSION) do |gz|
      gz.mtime = File.mtime(file)
      gz.orig_name = file
      gz.puts File.open(file, 'rb'){|f| f.read }
    end
    compressed_filename
  end

Run

bin/admin

migel> Migel::Util::Importer.new.reported_save_all_products('test.csv', 'de', true)
-> Array

Result

Email Tue Sep 20 10:32:21 2011: migel Migel::Util::Importer#reported_save_all_products(de)

Tue Sep 20 10:32:21 2011: migel Migel::Util::Importer#reported_save_all_products(de)
Saved file: /home/masa/ywesee/migel_dev/data/csv/test.csv
Total   571 Migelids (    3 Migelids have products /     1 Migelids have no products)
Saved 36 Products
Save time length: 40.93 [m]

Migelids with products (3)
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.01.00.1
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.02.00.2
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.02.02.2

Migelids without products (1)
http://ch.oddb.org/de/gcc/migel_search/migel_product/01.01.02.01.2

----

test.csv.gz	test.csv.gz
2K download

Note

Outputed file name should be renamed by an upper method to call the reported_save_all_products method
Exception process is not implemented at all

Testcases (keep in mind)

report
compresss
mail

Update sort faster

Problem

The following keywords search takes much long time
- 'SIGVARIS'
Probably it is included in both company name and product name

Task

Check the search process again
refactor it faster

There are two processes to search migel items

click the link -> ODDB::State::Global#migel_search method
use search box -> ODDB::State::Global#search method

Note

The first one search fast because a search method is selected by the url
The second one search slow because search all the items of migel sequentially

The search process (search box)

src/state/global.rb

      def search
...
          when :migel
            if result = @session.search_migel_products(query) and !result.empty?
              State::Migel::Result.new(@session, result)
            elsif result = @session.app.search_migel_items(query, @session.language) and !result.empty?
              product = StubProduct.new(result)
              sort_key = @session.user_input(:sortvalue) || @session.user_input(:reverse)
              reverse  = @session.user_input(:reverse)
              ODDB::State::Migel::Items.new(@session, ODDB::Migel::Items.new(product, sort_key, reverse))
            else
              State::Migel::Result.new(@session, [])
            end
...

Search order

@session.search_migel_products(query)
@session.app.search_migel_items(query, @session.language)

Experiment

src/state/global.rb

          when :migel
            if result = @session.search_migel_products(query) and !result.empty?
# masa
p "@session.search_migel_products(query)"
              State::Migel::Result.new(@session, result)
            elsif result = @session.app.search_migel_items(query, @session.language) and !result.empty?
p "@session.app.search_migel_items(query, @session.language)"
p "result.length = "
p result.length
              product = StubProduct.new(result)
p "done StubProduct.new"
              sort_key = @session.user_input(:sortvalue) || @session.user_input(:reverse)
              reverse  = @session.user_input(:reverse)
p "before view"
              ODDB::State::Migel::Items.new(@session, ODDB::Migel::Items.new(product, sort_key, reverse))
            else
              State::Migel::Result.new(@session, [])
            end

Search 'SIGVARIS'

http://oddb.masa.org/de/gcc/search/zone/migel/search_query/SIGVARIS#best_result

Note

This is the view problem because the result is too much

Solution

Show only first 200 records

Experiment

src/state/global.rb

          when :migel
            if result = @session.search_migel_products(query) and !result.empty?
              State::Migel::Result.new(@session, result)
            elsif result = @session.app.search_migel_items(query, @session.language) and !result.empty?
              #product = StubProduct.new(result)
              product = StubProduct.new(result[0,200])
              sort_key = @session.user_input(:sortvalue) || @session.user_input(:reverse)
              reverse  = @session.user_input(:reverse)
              ODDB::State::Migel::Items.new(@session, ODDB::Migel::Items.new(product, sort_key, reverse))
            else
              State::Migel::Result.new(@session, [])
            end

http://oddb.masa.org/de/gcc/search/zone/migel/search_query/SIGVARIS#best_result

Result

the result is shown quickly

efficient ODBA cache usage migel

Problem

The current system sends a request to migel server every time
The searching takes same time length every time
It seems that the migel cache is not efficiently used

Experiment

src/state/global.rb

      def migel_search
# masa
start_time = Time.now
p start_time
        if migel_code = @session.user_input(:migel_code) and product = @session.search_migel_products(migel_code).first
p "products.length = "
p product.products.length
          sort_key = @session.user_input(:sortvalue) || @session.user_input(:reverse)
          reverse  = @session.user_input(:reverse)
p "time = "
p Time.now - start_time
p "before view"
          ODDB::State::Migel::Items.new(@session, ODDB::Migel::Items.new(product, sort_key, reverse))

src/view/migel/items.rb

  def article_name(model = @model, session = @session)
p Time.now

http://oddb.masa.org/de/gcc/migel_search/migel_code/170202001

Result (Log)

1316519217
getin migel_search_by_migel_code
products.length = 567
time = 0.003927
before view
...
1316519225
...
1316519228
(end view)

Note

searching in migel server takes only 0.003 seconds
it takes about 11 seconds from the beginning of search and the end of showing the result
the slow of searching is because of 'view process' in ch.oddb.org side
it is NOT because of the migel cache process

Hypothesis

This may be by the transporting of objects called by reference

Solution

call by value
(fetch objects called by reference in the side of ch.oddb.org)

Experiment

src/model/migel/items.rb class Items def initialize(product, sortvalue = nil, reverse = nil)

    if product and items = product.items
      @sortvalue = sortvalue
      @list = items.values[0,200]

http://oddb.masa.org/de/gcc/migel_search/migel_code/170202001

Result (Log)

1316520431
"getin migel_search_by_migel_code"
"products.length = "
567
"time = "
0.005834
"before view"
1316520433
...
1316520435

Note

It takes 4 seconds
only 2 migelids have over 3000 products
- 17.03.02.00.1 (6920)
- 17.01.02.00.1 (3345)
to check the number of products
bin/admin

migel> open('/home/masa/work/test.dat','w'){|out| out.print migelids.values.map{|m| if products = m.products then m.migel_code + "\t" + products.length.to_s end}.join("\n")}

migel code list (over 300 products)

15.10.02.01.1   312
29.03.01.00.1   367
15.30.01.00.1   369
17.01.03.00.1   404
05.04.05.00.1   412
15.01.01.00.1   450
05.04.02.00.1   557
17.02.02.00.1   567
29.02.01.00.1   924
05.07.03.00.1   1051
29.01.01.00.1   1634
17.04.02.00.1   3293
17.01.02.00.1   3345
17.03.02.00.1   6920

Note

The paging function could help
For example, http://oddb.masa.org/de/gcc/search/zone/drugs/search_query/novartis/search_type/st_company#best_result

Paging function migel

Reference

src/view/pager.rb

Experiment

src/model/migel/items.rb

  def previous
    'previous'
  end
  def next
    'next'
  end

src/view/migel/items.rb

class SubHeader < HtmlGrid::Composite
..
  COMPONENTS = {
...
    [1,0]   => :pages,
...
  def pages(model, session=@session)
      pages = []
      args = {}
      View::Pager.new(pages, @session, self, :search, args)
  end

src/state/migel/items.rb

  def page
    page = ODDB::State::PageFacade.new(0)
    page.model = @model
  end

Access

http://oddb.masa.org/de/gcc/migel_search/migel_code/310409001

Result

Experiment

src/view/migel/items.rb

  def pages(model, session=@session)
      pages = @session.state.pages
      args = {}
      View::Pager.new(pages, @session, self, :search, args)
  end

src/state/migel/items.rb

class Items < State::Migel::Global
  attr_reader :pages
  def init
    @pages = []
    page = 0
    count = 0
    @model.each { |product|
      @pages[page] ||= ODDB::State::PageFacade.new(page)
      @pages[page].push(product)
      count += 1
      if(count >= 2)
        page += 1
        count = 0
      end
    }
  end
...
  def page
    page = @pages[0]
    page.model = @model
    page
  end

Access

http://oddb.masa.org/de/gcc/migel_search/migel_code/153050001

Result

Control the number of shown record
Create a link of the paging

Experiment

src/state/migel/items.rb

    def init
    @pages = []
    page = 0
    count = 0
    @model.each { |product|
      @pages[page] ||= ODDB::State::PageFacade.new(page)
      @pages[page].push(product)
      count += 1
      if(count >= 2)
        page += 1
        count = 0
      end
    }

    @filter = Proc.new { |model|
      page()
    }
    end

    def page
    page = @pages[1]
    page.model = @model
    page
    end

Access

http://oddb.masa.org/de/gcc/migel_search/migel_code/153050001

Result

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110920-report-paging-migel

Log reported_save_all_products migel

Check how to use ODBA cache data

Update report mail

List migel codes

Attach a file to report

Compress a file

Attach a compressed file

Update sort faster

efficient ODBA cache usage migel

Paging function migel

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110920-report-paging-migel

Log reported_save_all_products migel

Check how to use ODBA cache data

Update report mail

List migel codes

Attach a file to report

Compress a file

Attach a compressed file

Update sort faster

efficient ODBA cache usage migel

Paging function migel

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler