view · edit · sidebar · attach · print · history

20110815-migel-switch-source-data-oddb_org

<< | Index | >>

  1. Switch data structure from SearchedTable to Item class for the view of migel data
  2. Deactivate a link when the migel data is not found
  3. Import migel data
  4. Output migel data into a csv file
  5. Update Item class and import process for output csv
  6. Implement output process
  7. Debug search function
  8. Report process
Goal/Estimate/Evaluation
  • Migel function / 80% / 90%
Milestones
  • Switch the data for migel search function from SearchedTable class to Item class
  • save link information in the database (z.B. in ODDB::Migel::Product)
  • diactivate links
  • Refactoring
  • Import all the migel data
  • Output csv file
  • Debug search function
  • Report mail
  • Import and show fr migel data
Summary
Commits

Switch data structure from SearchedTable to Item class for the view of migel data

Confirm (Data) 

ch.oddb> ODDB::MiGeLPlugin.new(self).update_items_by_migel
-> Array
ch.oddb> search_migel_products('342101021','de').length
-> 1
ch.oddb> search_migel_products('342101021','de')[0]
-> Elastische (Ideal-)Binden, Fixation, 100 % Baumwolle, gedehnt, textilelastisch
ch.oddb> search_migel_products('342101021','de')[0].items.length
-> 7
ch.oddb> search_migel_products('342101021','de')[0].item('1740101').class
-> ODDB::Migel::Item
ch.oddb> search_migel_products('342101021','de')[0].item('1740101').pharmacode
-> 1740101
ch.oddb> search_migel_products('342101021','de')[0].item('1740101').pharmacode

Experiment

  • src/model/migel/searched_tabler.b 
require 'model/migel/item'
...
  def initialize(session, migel_code, migel_product)
#    plugin = ODDB::SwissindexNonpharmaPlugin.new(session.app)
#    @table = plugin.search_migel_table(migel_code).map do |record|
#      Record.new(record, self) 
#    end
    @table = migel_product.items.values
    @price = migel_product.price
    @qty = migel_product.qty
    @unit = migel_product.unit
    @pointer_descr = migel_product.migel_code
  end
  • src/model/migel/item.rb 
#!/usr/bin/env ruby
# ODDB::Migel::Item -- oddb.org -- 12.08.2011 -- mhatakeyama@ywesee.com

require 'util/language'
require 'model/text'

module ODDB
  module Migel
    class Item
      include SimpleLanguage
      attr_accessor :ean_code, :pharmacode, :article_name, :companyname, :ppha, :ppub, :factor, :size, :status
      attr_reader :product
      def initialize(product)
        @product = product
      end
      def price
        @product.price
      end
      def qty
        @product.qty
      end
      def unit
        @product.unit
      end
    end
  end
end

Run

  • bin/oddbd

Note

  • ext/swissindex/bin/swissindex_nonpharmad does not run

Access

Result

Access (for example)

Log 

error in SBSM::Session#process: /de/gcc/migel_search/migel_code/151002001
NoMethodError
undefined method `values' for nil:NilClass
/home/masa/ywesee/oddb.org/src/model/migel/searched_table.rb:42:in `initialize'
/home/masa/ywesee/oddb.org/src/state/global.rb:476:in `new'
/home/masa/ywesee/oddb.org/src/state/global.rb:476:in `migel_search'
/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:222:in `send'
/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:222:in `_trigger'

Note

  • The error comes because only the migel items data for 342101021 is saved in the database

Next

  • Diactivate a link when the migel data is not found

Deactivate a link when the migel data is not found

Experiment

  • src/model/migel/result.rb#migel_code 
  def migel_code(model)
    if items = model.items and !items.empty?
      link = PointerLink.new(:to_s, model, @session, self)
      link.value = model.migel_code
      link.href = @lookandfeel._event_url(:migel_search, {:migel_code => model.migel_code.gsub(/\./, '')})
      link
    else
      model.migel_code
    end
  end
  • src/view/migel/product.rb 
  def migel_code(model)
    if items = model.items and !items.empty?
      link = PointerLink.new(:to_s, model, @session, self)
      link.value = model.migel_code
      link.href = @lookandfeel._event_url(:migel_search, {:migel_code => model.migel_code.gsub(/\./, '')})
      link
    else
      model.migel_code
    end
  end

Access

Result

Next

  • Refactoring (SearchedTable class)

Note

  • Rename 'SearchedTable' to 'Items'

Import migel data

Experiment

  • src/plugin/migel.rb 
    def update_items_by_migel
      product = @app.search_migel_products('349903021', 'de')[0]
      migel_code = product.migel_code.split('.').to_s
      plugin = ODDB::SwissindexNonpharmaPlugin.new(@app)
      if table = plugin.search_migel_table(migel_code)
        table.each do |record|
          if record[:pharmacode]
            update_item(product, record)
          end
        end
      end
    end

Run

  • bin/oddbd
  • ext/swissindex/bin/swissindex_nonpharmad
  • bin/admin 
ch.oddb> ODDB::MiGeLPlugin.new(self).update_items_by_migel
-> Array

Result

Before

After

Next

  • Import all the migel data
  • Calculate the estimated importing time length

Experiment

  • src/plugin/migel.rb 
    def estimate_time(start_time, total, count)
      estimate = (Time.now - start_time) * total / count
      log = count.to_s + " / " + total.to_s + "\t"
      em   = estimate/60
      eh   = em/60
      rest = estimate - (Time.now - start_time)
      rm   = rest/60
      rh   = rm/60
      log << "Estimate total: "
      if eh > 1.0
        log << "%.2f" % eh + " [h]"
      else
        log << "%.2f" % em + " [m]"
      end
      log << " It will be done in: "
      if rh > 1.0
        log << "%.2f" % rh + " [h]\n"
      else
        log << "%.2f" % rm + " [m]\n"
      end
      log
    end
    def update_items_by_migel(time_estimate = false)
      total = @app.migel_count
      start_time = Time.now
      @app.migel_products.each_with_index do |product, count|
        migel_code = product.migel_code.split('.').to_s
        plugin = ODDB::SwissindexNonpharmaPlugin.new(@app)
        if table = plugin.search_migel_table(migel_code)
          table.each do |record|
            if record[:pharmacode]
              update_item(product, record)
            end
          end
        end
        p estimate_time(start_time, total, count + 1) if time_estimate
      end
    end

Run 

ch.oddb> ODDB::MiGeLPlugin.new(self).update_items_by_migel true

Log 

"1 / 571\tEstimate total: 1.56 [h] It will be done in: 1.56 [h]\n"
"2 / 571\tEstimate total: 57.60 [m] It will be done in: 57.40 [m]\n"
"3 / 571\tEstimate total: 42.62 [m] It will be done in: 42.40 [m]\n"
"4 / 571\tEstimate total: 1.10 [h] It will be done in: 1.09 [h]\n"
"5 / 571\tEstimate total: 53.83 [m] It will be done in: 53.36 [m]\n"
"6 / 571\tEstimate total: 45.18 [m] It will be done in: 44.71 [m]\n"
"7 / 571\tEstimate total: 38.97 [m] It will be done in: 38.49 [m]\n"
"8 / 571\tEstimate total: 34.40 [m] It will be done in: 33.92 [m]\n"
"9 / 571\tEstimate total: 1.90 [h] It will be done in: 1.87 [h]\n"
"10 / 571\tEstimate total: 1.75 [h] It will be done in: 1.72 [h]\n"
"11 / 571\tEstimate total: 1.60 [h] It will be done in: 1.57 [h]\n"
"12 / 571\tEstimate total: 1.47 [h] It will be done in: 1.44 [h]\n"
"13 / 571\tEstimate total: 1.55 [h] It will be done in: 1.51 [h]\n"
"14 / 571\tEstimate total: 1.46 [h] It will be done in: 1.42 [h]\n"
"15 / 571\tEstimate total: 1.38 [h] It will be done in: 1.34 [h]\n"
"16 / 571\tEstimate total: 1.39 [h] It will be done in: 1.35 [h]\n"
"17 / 571\tEstimate total: 1.34 [h] It will be done in: 1.30 [h]\n"
"18 / 571\tEstimate total: 2.15 [h] It will be done in: 2.08 [h]\n"
"19 / 571\tEstimate total: 2.39 [h] It will be done in: 2.31 [h]\n"
"20 / 571\tEstimate total: 2.71 [h] It will be done in: 2.61 [h]\n"
"21 / 571\tEstimate total: 2.63 [h] It will be done in: 2.53 [h]\n"
"22 / 571\tEstimate total: 2.55 [h] It will be done in: 2.45 [h]\n"
"23 / 571\tEstimate total: 2.49 [h] It will be done in: 2.39 [h]\n"
"24 / 571\tEstimate total: 2.45 [h] It will be done in: 2.34 [h]\n"
"25 / 571\tEstimate total: 2.38 [h] It will be done in: 2.27 [h]\n"
...

Note

  • It will take a few hours to import all the migel data

Error 

...
"161 / 571\tEstimate total: 2.25 [h] It will be done in: 1.62 [h]\n"
"162 / 571\tEstimate total: 2.32 [h] It will be done in: 1.66 [h]\n"
/usr/lib64/ruby/site_ruby/1.8/odba/persistable.rb:577: [BUG] Segmentation fault
ruby 1.8.6 (2009-06-08) [x86_64-linux]

Abgebrochen

Commit

Output migel data into a csv file

Experiment 

ch.oddb> search_migel_products('349903021','de')[0].oid
-> 
ch.oddb> search_migel_products('349903021','de')[0].oid.class
-> NilClass
ch.oddb> search_migel_products('349903021','de')[0].odba_id
-> 797294

Note

  • 'oid' method is not available (I do not know why)
  • I should use 'odba_id' instead of 'oid'
  • They are same. See: src/util/persistence.rb

Design

  • There is alread a method, migel_nonpharma(pharmacode_file), which calls search_migel(pharmacode), search_migel_position_number(pharmacode), and search_itme(pharmacode) methods
  • In this time, pharmacode list comes from from migel_search_table(migel_code) method
  • The simplest solution is as follows:
    1. call search_migel_table(migel_code) and get pharmacode list (Array)
    2. call search_migel(pharmacode), search_migel_position_number(pharmacode), and search_itme(pharmacode) in the iteration block of pharmacode list array

Note

  • But the calling of search_migel_table, search_migel, and search_migel_position_number is not efficient but redundant.
  • -> create a new method to output one line data from one pharmacode for the output csv file.
  • -> import migel data frist, and then output csv file second in the pharmacode iteration

Next

  • Refactoring plugin/swissindex.rb
  • Update Item class to be able to save all the data for output csv file
  • Confirm import

Update Item class and import process for output csv

Experiment

  • ext/swissindex/src/swissindex.rb 
  def merge_swissindex_migel(swissindex_item, migel_line)
    # Swissindex data
    swissindex = {}
    swissindex.update({
      :ean_code => swissindex_item[:gtin],
      :datetime => swissindex_item[:dt],
      :stdate   => swissindex_item[:stdate],
      :language => swissindex_item[:lang],
      :article_name => swissindex_item[:dscr],
      :size     => swissindex_item[:addscr],
      :status   => swissindex_item[:status],
    })
    if company = swissindex_item[:comp]
      swissindex[:companyname] = company[:name]
      swissindex[:companyean]  = company[:gln]
    end

    # Migel data
    pharmacode, article_name, companyname, ppha, ppub, factor = *migel_line
    migel = {
      :pharmacode   => pharmacode,
      :article_name => article_name,
      :companyname  => companyname,
      :ppha         => ppha,
      :ppub         => ppub,
      :factor       => factor,
    }
    migel.update swissindex
  end

  def search_migel_table(code, query_key = 'Pharmacode')
    # 'MiGelCode' is also available for query_key
    agent = Mechanize.new
    try_time = 3
    begin
      agent.get(@base_url + query_key + '=' + code)
      count = 100
      table = []
      line  = []
      migel = {}
      agent.page.search('td').each_with_index do |td, i|
        text = td.inner_text.chomp.strip
        if text.is_a?(String) && text.length == 7 && text.match(/\d{7}/)
          migel_item = if pharmacode = line[0] and pharmacode.match(/\d{7}/) and swissindex_item = search_item(pharmacode)
                         merge_swissindex_migel(swissindex_item, line)
                       else
                         merge_swissindex_migel({}, line)
                       end
          table << migel_item
          line = []
          count = 0
        end
        if count < 7
          text = text.split(/\n/)[1] || text.split(/\n/)[0]
          text = text.gsub(/\302\240/, '').strip if text
          line << text
          count += 1
        end
      end

      # for the last line
      migel_item = if pharmacode = line[0] and pharmacode.match(/\d{7}/) and swissindex_item = search_item(pharmacode)
                     merge_swissindex_migel(swissindex_item, line)
                   else
                     merge_swissindex_migel({}, line)
                   end
      table << migel_item
      table.shift
      table
  • src/plugin/migel.rb 
  #def update_items_by_migel(time_estimate = false)
  def update_items_by_migel(migel_code)
      total = @app.migel_count
      start_time = Time.now
      #@app.migel_products.each_with_index do |product, count|
      product = search_migel_products(migel_code, 'de')[0]
        migel_code = product.migel_code.split('.').to_s
        plugin = ODDB::SwissindexNonpharmaPlugin.new(@app)
        if table = plugin.search_migel_table(migel_code)
          table.each do |record|
            if record[:pharmacode]
              update_item(product, record)
            end
          end
        end
#        p estimate_time(start_time, total, count + 1) if time_estimate
#      end
    end

    def update_item(product, record)
      pointer = product.pointer + [:item, record[:pharmacode]]
      update_values = {
        :pharmacode   => record[:pharmacode],
        :ean_code     => record[:ean_code],
        :article_name => record[:article_name],
        :companyname  => record[:companyname],
        :ppha         => record[:ppha],
        :ppub         => record[:ppub],
        :factor       => record[:factor],
        :pzr          => record[:pzr],
        :size         => record[:size],
        :status       => record[:status],
        :datetime     => record[:datetime],
        :stdate       => record[:stdate],
        :language     => record[:language],
      } 
      @app.update(pointer.creator, update_values, :migel)
    end
  • src/model/migel/item.rb 
#!/usr/bin/env ruby
# encondig: utf-8
# ODDB::Migel::Item -- oddb.org -- 15.08.2011 -- mhatakeyama@ywesee.com

require 'util/language'
require 'model/text'

module ODDB
  module Migel
    class Item
      include SimpleLanguage
      attr_accessor :ean_code, :pharmacode, :article_name, :companyname, :ppha, :ppub, 
        :factor, :pzr, :size, :status, :datetime, :stdate, :language,
...

Run

  • bin/admin 
ch.oddb> ODDB::MiGeLPlugin.new(self).update_items_by_migel '342101021'
-> Array

Result 

ch.oddb> search_migel_products('342101021','de')[0].item('1740101').pzr
-> 5.60

Implement output process

  • src/plugin/swissindex.rb 
  class SwissindexNonpharmaPlugin < SwissindexPlugin
    SWISSINDEX_NONPHARMA_SERVER = DRbObject.new(nil, ODDB::Swissindex::SwissindexNonpharma::URI)
    def migel_nonpharma_one_line(migel_item)
      [
        migel_item.odba_id,
        migel_item.migel_code,
        migel_item.pharmacode,
        migel_item.ean_code,
        migel_item.datetime,
        migel_item.status,
        migel_item.stdate,
        migel_item.language,
        migel_item.article_name,
        migel_item.size,
        migel_item.companyname,
        migel_item.company_ean,
        migel_item.ppha,
        migel_item.pppub,
        migel_item.factor,
        migel_item.pzr,
      ].join(";") 
    end
    def output_migel_nonpharma
      dir = File.expand_path('../../data/csv', File.dirname(__FILE__))
      FileUtils.mkdir_p dir
      output_file = File.join(dir, 'swissINDEX_MiGel.csv')
      open(output_file, "w") do |out|
        f.print "odba_id;position number;pharmacode;GTIN;datetime;status;stdate;lang;description;additional description;company name;company GLN;pharmpreis;ppub;faktor;pzr\n" 
        @app.each_migel_product do |product|
          product.items.each do |item|
            out.print migel_nonpharma_one_line(item), "\n"
          end                                
        end
      end
    end

Run

  • bin/admin 
ch.oddb> ODDB::SwissindexNonpharmaPlugin.new(self).output_migel_nonpharma
-> Hash

Result

  • success

Debug search function

Bug

  • Search function fails sometime in migel

Then

Log 

error in SBSM::Session#to_html: /de/gcc/search/zone/migel/search_query/151002011
NoMethodError
undefined method `items' for #<ODDB::Migel::Group:0x7fb0ea61b6b8>
/home/masa/ywesee/oddb.org/src/util/language.rb:44:in `method_missing'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:83:in `migel_code'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:113:in `compose_subheader'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:100:in `compose_list'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:97:in `each'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:97:in `compose_list'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/list.rb:54:in `compose'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:55:in `init'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/list.rb:129:in `init'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:59:in `init'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/component.rb:138:in `initialize'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:59:in `new'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:59:in `create'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:280:in `compose_component'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:209:in `compose'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:203:in `each'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:203:in `compose'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:55:in `init'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/component.rb:138:in `initialize'
/home/masa/ywesee/oddb.org/src/view/publictemplate.rb:53:in `new'
/home/masa/ywesee/oddb.org/src/view/publictemplate.rb:53:in `content'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:66:in `send'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:66:in `create'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:280:in `compose_component'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:209:in `compose'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:203:in `each'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:203:in `compose'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:55:in `init'
/home/masa/ywesee/oddb.org/src/view/publictemplate.rb:50:in `init'
/home/masa/ywesee/oddb.org/src/view/privatetemplate.rb:16:in `init'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/component.rb:138:in `initialize'
/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:285:in `new'
/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:285:in `view'
/usr/lib64/ruby/site_ruby/1.8/sbsm/state.rb:197:in `to_html'
/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:529:in `to_html'
/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:177:in `drb_process'
/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:174:in `synchronize'
/usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb:174:in `drb_process'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1515:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1589:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1430:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1347:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `start_service'
bin/oddbd:38
ODDB::View::Migel::ResultComposite::COMPONENTS[[0, 0]] in create(ODDB::View::Migel::ResultList)
ODDB::View::Migel::Result::COMPONENTS[[0, 3]] in create(content)
error in SBSM::Session#http_headers: /de/gcc/search/zone/migel/search_query/151002011
NoMethodError
undefined method `items' for #<ODDB::Migel::Group:0x7fb0ea61b6b8>
/home/masa/ywesee/oddb.org/src/util/language.rb:44:in `method_missing'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:83:in `migel_code'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:113:in `compose_subheader'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:100:in `compose_list'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:97:in `each'

Experiment

  • src/view/migel/result.rb 
   def migel_code(model)
    if model.respond_to?(:items) and items = model.items and !items.empty?
      link = PointerLink.new(:to_s, model, @session, self)
      link.value = model.migel_code
      link.href = @lookandfeel._event_url(:migel_search, {:migel_code => model.migel_code.gsub(/\./, '')})
      link
    else
      model.migel_code
    end
   end

Result

  • success

Report process

  • src/util/updater.rb 
    def export_migel_nonpharma
      update_notify_simple(SwissindexNonpharmaPlugin, 'Export Swissindex Migel Nonpharma', :export_migel_nonpharma)
    end
  • src/plugin/swissindex.rb 
    def export_migel_nonpharma
      dir = File.expand_path('../../data/csv', File.dirname(__FILE__))
      FileUtils.mkdir_p dir
      @output_file = File.join(dir, 'swissINDEX_MiGel.csv')
      open(@output_file, "w") do |out|
        out.print "odba_id;position number;pharmacode;GTIN;datetime;status;stdate;lang;description;additional description;company name;company GLN;pharmpreis;ppub;faktor;pzr\n"
        @app.each_migel_product do |product|
          if items = product.items
            items.values.each do |item|
              out.print migel_nonpharma_one_line(item), "\n"
            end
          end
        end
      end
    end

Run

  • bin/admin 
ch.oddb> Updater.new(self).export_migel_nonpharma
-> mhatakeyama@ywesee.com

Result

Note

  • But this is 'export' not 'update'

Commit

view · edit · sidebar · attach · print · history
Page last modified on August 15, 2011, at 05:02 PM