view · edit · sidebar · attach · print · history

20110812-migel-import-oddb_org

<< | Index | >>

Design Item class for migel data
Build up Item class
Update search_migel_table method to get ean code too
Take article_name, companyname, status, size from swissindex
Update import process
Switch data structure from SearchedTable to Item class for the view of migel data

Goal/Estimate/Evaluation

Migel import function / 80% / 80%

Milestones

~~develop ODDB::Migel::Item class and save the instance in the database via ODBA~~
~~Get ean code in search_migel_table method~~
~~Take the article name, companyname, status, size from swissindex, not migel~~
Update import process and Item class
Switch the data for migel search function from SearchedTable class to Item class
import function
save link information in the database (z.B. in ODDB::Migel::Product)
diactivate links

Summary

http://masay.ch/2011/08/12/updated-migel-import-process-in-oddb-org/

Commits
ToDo Tomorrow

Design Item class for migel data

Experiment

src/plugin/migel.rb

    def update_item(product)
      pointer = product.pointer + [:item]
      @app.update(pointer.creator, {}, :migel)
    end

Run

bin/oddbd
bin/admin

ch.oddb> search_migel_products('151002001','de')[0].pointer
-> :!migel_group,15!subgroup,10!product,02%.00%.1.
ch.oddb> ODDB::MiGeLPlugin.new(self).update_item(search_migel_products('151002001','de')[0])
-> undefined method `diff' for #<ODDB::Persistence::InvalidPathError:0x7f16647d9138>

Note

ODDB::Persistence::InvalidPathError comes

Experiment

src/plugin/migel.rb

    def update_test1
      group_pointer = Persistence::Pointer.new([:migel_group, '111'])
      @app.update(group_pointer.creator, {}, :migel)
    end
    def update_test2
      group_pointer = Persistence::Pointer.new([:masa, '222'])
      @app.update(group_pointer.creator, {}, :migel)
    end

Result

ch.oddb> migel_group('111').class
-> ODDB::Migel::Group
ch.oddb> ODDB::MiGeLPlugin.new(self).update_test2
-> undefined method `diff' for #<ODDB::Persistence::InvalidPathError:0x7fc60ec96bf8>
ch.oddb> migel_group('222').class
-> NilClass

Consideration

'migel_group' in the argument of Pointer.new may be related to ODDB::Migel::Group class

Experiment

src/model/migel/item.rb
1. !/usr/bin/env ruby
2. ODDB::Migel::Item -- oddb.org -- 12.08.2011 -- mhatakeyama@ywesee.com
require 'util/language' require 'model/text'
module ODDB module Migel class Item

      include SimpleLanguage
      attr_accessor :pharmacode
      attr_reader :product
      def initialize(product)
        @product = product
      end
    end

end

end

src/model/migel/product.rb module ODDB module Migel class Product

      attr_reader :code, :accessories, :products, :product_text, :items

...

      def initialize(code)
  ...
        @items = {}
      end
      def create_item(pharmacode)
        @items ||= {}
        item = ODDB::Migel::Item.new(self)
        @items.store(pharmacode, item)
        item
      end
      def item(pharmacode)
        @items[pharmacode]
      end

src/plugin/migel.rb

    def update_test
      product = @app.search_migel_products('151002011', 'de')[0]
      pointer = product.pointer + [:item, '999']
      update_values = {
        :pharmacode => '1234'
      }
      @app.update(pointer.creator, update_values, :migel)
    end

Result

ch.oddb> search_migel_products('151002011', 'de')[0].item('999').class
-> NilClass
ch.oddb> ODDB::MiGeLPlugin.new(self).update_test
-> 
ch.oddb> search_migel_products('151002011', 'de')[0].item('999').class
-> ODDB::Migel::Item
ch.oddb> search_migel_products('151002011', 'de')[0].item('999').pharmacode
-> 1234

Note

Pointer data structure and class structure is closely related
create_item and item() methods are important and these are called from @app.update automatically
@app.update automatically create an instance
Pointer [:item, '999']: the item is used for the method names, 'create_item' and 'item' in Product class
The 999 becomes an argument of 'create_method'
update_values = {:pharmacode => '1234'}: pharmacode becomes an accessor of Item class, the 1234 becomes an actual value, namely, Item#pharmacode='1234' is called from @app.update method
'delete_xxx' and 'checkout' methods in Product class probably should be updated for Item class too.

Build up Item class

Build up Item class

Experiment

src/model/migel/item.rb

#!/usr/bin/env ruby
# ODDB::Migel::Item -- oddb.org -- 12.08.2011 -- mhatakeyama@ywesee.com

require 'util/language'
require 'model/text'

module ODDB
  module Migel
    class Item
      include SimpleLanguage
      attr_accessor :pharmacode, :article_name, :companyname, :ppha, :ppub, :factor
      attr_reader :product
      def initialize(product)
        @product = product
      end
    end
  end
end

src/plugin/migel.rb

    def update_items_by_migel
      #product = @app.migel_products[0]
      product = @app.search_migel_products('151002011', 'de')[0]
      migel_code = product.migel_code.split('.').to_s
      plugin = ODDB::SwissindexNonpharmaPlugin.new(@app)
      if table = plugin.search_migel_table(migel_code)
        table.each do |record|
          if pharmacode = record.shift
            update_item(product, pharmacode, *record)
          end
        end
      end
    end
    def update_item(product, pharmacode, *args)
      #product = @app.search_migel_products('151002011', 'de')[0]
      pointer = product.pointer + [:item, pharmacode]
      article_name, companyname, ppha, ppub, factor = *args
      update_values = {
        :pharmacode   => pharmacode,
        :article_name => article_name,
        :companyname  => companyname,
        :ppha         => ppha,
        :ppub         => ppub,
        :factor       => factor,
      }
      @app.update(pointer.creator, update_values, :migel)
    end

Run

bin/oddbd
ext/swissindex/bin/swissindex_nonpharmad
bin/admin

ch.oddb> ODDB::MiGeLPlugin.new(self).update_items_by_migel

Result

ch.oddb> search_migel_products('151002011', 'de')[0].item('4340649').pharmacode
-> 4340649
ch.oddb> search_migel_products('151002011', 'de')[0].item('4340649').companyname
-> Manfred
ch.oddb> search_migel_products('151002011', 'de')[0].item('4340649').ppha
-> 100.95
ch.oddb> search_migel_products('151002011', 'de')[0].item('4340649').ppub
-> 134.25
ch.oddb> search_migel_products('151002011', 'de')[0].item('4340649').factor
-> 30
ch.oddb> search_migel_products('151002011', 'de')[0].items.keys.length
-> 323

Note (Important)

if an item is deleted by ODDB::Migel::Product.delete_item(pharmacode), the pharmacode number will never be used
If I execute update_items_by_migel again after the deletion, the item is not saved in the database

Question

What does the odba_isorate_store method do? suspend

Update search_migel_table method to get ean code too

Experiment

src/custom/lookandfeelbase.rb

        :th_ean_code              =>  'EAN-Code',

src/model/migel/searched_table.rb
class SearchedTable class Record

      attr_reader :ean_code, :pharmacode, :article_name, :companyname, :ppha, :ppub, :factor
      def initialize(record, container)
        @ean_code, @pharmacode, @article_name, @companyname, @ppha, @ppub, @factor = *record

src/view/migel/searched_table.rb

class SearchedList < HtmlGrid::List
  CSS_CLASS = 'composite'
  SUBHEADER = ODDB::View::Migel::SubHeader
  def init
    @components = {
      [0,0]   =>  :pharmacode,
      [1,0]   =>  :ean_code,
      [2,0]   =>  :article_name,
      [3,0]   =>  :companyname,
      [4,0]   =>  :ppha,
      [5,0]   =>  :ppub,
      [6,0]   =>  :factor,
    }
    @css_map = {
      [0,0]   => 'list',
      [1,0]   => 'list',
      [2,0]   => 'list bold',
      [3,0]   => 'list italic',
      [4,0]   => 'list',
      [5,0]   => 'list',
      [6,0]   => 'list',
    }
    super
  end

Result

Note

~~I should update this to show the ean-code as mouse over window later~~ canceled
-> keep this view

Take the article name from swissindex, not migel

Take article_name, companyname, status, size from swissindex

Experiemnt

src/custom/lookandfeelbase.rb

        :th_ean_code              =>  'EAN-Code',
        :th_article_name          =>  'Artikelname',
        :th_companyname           =>  'Companyname',
        :th_ppha                  =>  'PPHA',
        :th_ppub                  =>  'PPUB',
        :th_factor                =>  'Faktor',
        :th_status                =>  'Status',

src/view/migel/searched_table.rb

class SearchedList < HtmlGrid::List
  CSS_CLASS = 'composite'
  SUBHEADER = ODDB::View::Migel::SubHeader
  def init
    @components = {
      [0,0]   =>  :pharmacode,
      [1,0]   =>  :ean_code,
      [2,0]   =>  :article_name,
      [3,0]   =>  :size,
      [4,0]   =>  :status,
      [5,0]   =>  :companyname,
      [6,0]   =>  :ppha,
      [7,0]   =>  :ppub,
      [8,0]   =>  :factor,
    }
    @css_map = {
      [0,0]   => 'list',
      [1,0]   => 'list',
      [2,0]   => 'list bold',
      [3,0]   => 'list italic',
      [4,0]   => 'list',
      [5,0]   => 'list',
      [6,0]   => 'list',
      [7,0]   => 'list',
      [8,0]   => 'list',
    }
    super
  end

src/model/migel/searched_table.rb

class SearchedTable
  class Record
    attr_reader :ean_code, :pharmacode, :article_name, :companyname, :ppha, :ppub, :factor, :size, :status
    def initialize(record, container)
      @ean_code     = record[:ean_code]
      @pharmacode   = record[:pharmacode]
      @article_name = record[:article_name]
      @companyname  = record[:companyname]
      @ppha         = record[:ppha]
      @ppub         = record[:ppub]
      @factor       = record[:factor]
      @size         = record[:size]
      @status       = record[:status]
      @container    = container
    end

ext/swissindex/src/swissindex.rb

  def search_migel_table(code, query_key = 'Pharmacode')
    # 'MiGelCode' is also available for query_key
    agent = Mechanize.new
    try_time = 3
    begin
      agent.get(@base_url + query_key + '=' + code)
      count = 100
      table = []
      line  = []
      migel = {}
      agent.page.search('td').each_with_index do |td, i|
        text = td.inner_text.chomp.strip
        if text.is_a?(String) && text.length == 7 && text.match(/\d{7}/)
          swissindex = {}
          if pharmacode = line[0] and pharmacode.match(/\d{7}/) and item = search_item(pharmacode)
            swissindex[:ean_code] = item[:gtin]
            swissindex[:article_name] = item[:dscr]
            swissindex[:size] = item[:addscr]
            swissindex[:status] = item[:status]
            if company = item[:comp]
              swissindex[:companyname] = company[:name]
            end
          end

          pharmacode, article_name, companyname, ppha, ppub, factor = *line
          migel = {
            :pharmacode   => pharmacode,
            :article_name => article_name,
            :companyname  => companyname,
            :ppha         => ppha,
            :ppub         => ppub,
            :factor       => factor,
          }
          migel.update swissindex
#          line.unshift ean_code
#          table << line
          table << migel
          line = []
          swisindex = {}
          count = 0
        end
        if count < 7
          text = text.split(/\n/)[1] || text.split(/\n/)[0]
          text = text.gsub(/\302\240/, '').strip if text
          line << text
          count += 1
        end
      end
      # for the last line
      swissindex = {}
      if pharmacode = line[0] and pharmacode.match(/\d{7}/) and item = search_item(pharmacode)
        swissindex[:ean_code] = item[:gtin]
        swissindex[:article_name] = item[:dscr]
        swissindex[:size] = item[:addscr]
        swissindex[:status] = item[:status]
        if company = item[:comp]
          swissindex[:companyname] = company[:name]
        end
      end
      pharmacode, article_name, companyname, ppha, ppub, factor = *line
      migel = {
        :pharmacode   => pharmacode,
        :article_name => article_name,
        :companyname  => companyname,
        :ppha         => ppha,
        :ppub         => ppub,
        :factor       => factor,
      }
      migel.update swissindex

      table << migel
=begin
      ean_code = nil
      if pharmacode = line[0] and pharmacode.match(/\d{7}/) 
        ean_code = search_item(pharmacode)[:gtin]
      end
      line.unshift ean_code
      table << line
=end

      table.shift
      table
    rescue StandardError, Timeout::Error => err
      if try_time > 0
        puts err
        puts err.backtrace
        puts
        puts "retry"
        sleep 10
        agent = Mechanize.new
        try_time -= 1
        retry
      else
        return []
      end
    end
  end

Result

Note

search_migel_table method should be refactored
some code are doubled in one method

Update Item class structure

Update import process

Experiment

src/model/migel/item.rb

#!/usr/bin/env ruby
# ODDB::Migel::Item -- oddb.org -- 12.08.2011 -- mhatakeyama@ywesee.com

require 'util/language'
require 'model/text'

module ODDB
  module Migel
    class Item
      include SimpleLanguage
      attr_accessor :ean_code, :pharmacode, :article_name, :companyname, :ppha, :ppub, :factor, :size, :status
      attr_reader :product
      def initialize(product)
        @product = product
      end
    end
  end
end
~

src/plugin/migel.rb

    def update_items_by_migel
      #product = @app.migel_products[0]
      product = @app.search_migel_products('342101021', 'de')[0]
      migel_code = product.migel_code.split('.').to_s
      plugin = ODDB::SwissindexNonpharmaPlugin.new(@app)
      if table = plugin.search_migel_table(migel_code)
        table.each do |record|
          if record[:pharmacode]
            update_item(product, record)
          end
#          if pharmacode = record.shift
#            update_item(product, pharmacode, *record)
#          end
        end
      end
    end
    #def update_item(product, pharmacode, *args)
    def update_item(product, record)
      #product = @app.search_migel_products('151002011', 'de')[0]
      pointer = product.pointer + [:item, record[:pharmacode]]
#      article_name, companyname, ppha, ppub, factor = *args
      update_values = {
        :pharmacode   => record[:pharmacode],
        :ean_code     => record[:ean_code],
        :article_name => record[:article_name],
        :companyname  => record[:companyname],
        :ppha         => record[:ppha],
        :ppub         => record[:ppub],
        :factor       => record[:factor],
        :size         => record[:size],
        :status       => record[:status],
      }
      @app.update(pointer.creator, update_values, :migel)
    end

Result

ch.oddb> ODDB::MiGeLPlugin.new(self).update_items_by_migel
-> Array
ch.oddb> search_migel_products('342101021', 'de')[0].items.length
-> 7
ch.oddb> search_migel_products('342101021', 'de')[0].items('1740101')
-> (eval):1:in `items': wrong number of arguments (1 for 0)
ch.oddb> search_migel_products('342101021', 'de')[0].item('1740101')
-> 
ch.oddb> search_migel_products('342101021', 'de')[0].item('1740101').class
-> ODDB::Migel::Item
ch.oddb> search_migel_products('342101021', 'de')[0].item('1740101').status
-> A
ch.oddb> search_migel_products('342101021', 'de')[0].item('1740101').size
-> 10 Stk
ch.oddb> search_migel_products('342101021', 'de')[0].item('1740101').article_name
-> RHENA IDEAL Elastische Binde �cru 5mx6cm

Switch the data from SearchedTable to Item class

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110812-migel-import-oddb_org

Design Item class for migel data

Build up Item class

Update search_migel_table method to get ean code too

Take article_name, companyname, status, size from swissindex

Update import process

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110812-migel-import-oddb_org

Design Item class for migel data

Build up Item class

Update search_migel_table method to get ean code too

Take article_name, companyname, status, size from swissindex

Update import process

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler