20110818-import-drb-connection-migel-drb-server

<< | Index | >>

1. Import migel data
   1. Migel::Model::Group
   2. Migel::Model::Subroup
   3. Migel::Model::Migelid
   4. Migel::Model::Product
2. Wrapper class of migel data in oddb.org

Goal/Estimate/Evaluation

• Migel drb server data import / 80% / 90%

Milestones

• Import data
• Show migel data
  1. wrap class

Summary

• http://masay.ch/2011/08/18/develop-import-process-in-migel-drb-server/

Commits

ToDo Tomorrow

• prune_old_revision
• The other language (fr)
• Migel::Util::Swissindex.search_migel_table(pharmacode, language)
• Refactoring (in particluar, attr_accessor)
• Testcases oddb.org

Import migel data

Memo (vim, paste, cancel indent)

 :a!
 (paste)
 ESC

Plan

1. Import basic data from csv file
   • Refer to: http://dev.ywesee.com/wiki.php/Choddb/MiGel
   • src/util/updater.rb#update_migel
   • src/plugin/migel.rb
2. Import migel product data from online

Migel::Model::Group

Experiment (test only Migel::Model::Group)

• lib/migel/model/group.rb

#!/usr/bin/env ruby
# encoding: utf-8
# Migel::Model::Group -- migel -- 18.08.2011 -- mhatakeyama@ywesee.com

#$: << File.expand_path("../../../src", File.dirname(__FILE__))

module Migel
  module Model
    class Group < Migel::ModelSuper
      attr_accessor :limitation_text
      attr_reader :code
      has_many :subgroups, on_delete(:cascade), on_save(:cascade)
      multilingual :limitation_text
      multilingual :name
      alias :pointer_descr :code
      def initialize(groupcd)
        @code = groupcd
      end
    end
  end
end

• lib/migel/importer.rb

#!/usr/bin/env ruby
# enconding: utf-8
# Migel::Importer -- migel -- 18.08.2011 -- mhatakeyama@ywesee.com

require 'csv'

module Migel
  class Importer
    def update(path, language)
      CSV.readlines(path)[1..-1].each do |row|
        id = row.at(13).to_s.split('.')
        if(id.empty?)
          id = row.at(4).to_s.split('.')
        else
          id[-1].replace(id[-1][0,1])
        end
        unless id.empty?
          group = update_group(id, row, language)
          p group
        end
      end
    end
    def update_group(id, row, language)
      groupcd = id.at(0)
      hash = {
        :code => groupcd,
        language  => row.at(2).to_s,
      }
      group = Migel::Model::Group.find_by_code(groupcd) || Migel::Model::Group.new(groupcd)
      group.name.send(language.to_s + '=', row.at(2).to_s)
      text = row.at(3).to_s
      text.tr!("\v", " ")
      text.strip!
      unless(text.empty?)
        group.limitation_text.send(language.to_s + '=', text)
      end
      group.save
      group
    end
  end
end

• lib/migel/util/server.rb

require 'migel/importer'

Run

• bin/migeld
• bin/admin

migel> Migel::Importer.new.update('data/csv/migel_de_sample.csv','de')
-> Array

Log

#<Migel::Model::Group:0x7f9d13903438 @odba_id=5, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d13903320 @canonical={:de=>"ABSAUGGERAETE"}, @synonyms=[]>, @odba_observers=[], @code="01">
#<Migel::Model::Group:0x7f9d13903438 @odba_id=5, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d13903320 @canonical={:de=>"ABSAUGGERAETE"}, @synonyms=[]>, @odba_observers=[], @code="01">
#<Migel::Model::Group:0x7f9d13903438 @odba_id=5, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d13903320 @canonical={:de=>"ABSAUGGERAETE"}, @synonyms=[]>, @odba_observers=[], @code="01">
#<Migel::Model::Group:0x7f9d13903438 @odba_id=5, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d13903320 @canonical={:de=>"ABSAUGGERAETE"}, @synonyms=[]>, @odba_observers=[], @code="01">
#<Migel::Model::Group:0x7f9d13903438 @odba_id=5, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d13903320 @canonical={:de=>"ABSAUGGERAETE"}, @synonyms=[]>, @odba_observers=[], @code="01">
#<Migel::Model::Group:0x7f9d13903438 @odba_id=5, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d13903320 @canonical={:de=>"ABSAUGGERAETE"}, @synonyms=[]>, @odba_observers=[], @code="01">
#<Migel::Model::Group:0x7f9d138529d0 @odba_id=7, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d138528e0 @canonical={:de=>"APPLIKATIONSHILFEN"}, @synonyms=[]>, @odba_observers=[], @code="03">
#<Migel::Model::Group:0x7f9d138529d0 @odba_id=7, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d138528e0 @canonical={:de=>"APPLIKATIONSHILFEN"}, @synonyms=[]>, @odba_observers=[], @code="03">
#<Migel::Model::Group:0x7f9d138529d0 @odba_id=7, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d138528e0 @canonical={:de=>"APPLIKATIONSHILFEN"}, @synonyms=[]>, @odba_observers=[], @code="03">
#<Migel::Model::Group:0x7f9d138529d0 @odba_id=7, @subgroups=[], @odba_persistent=true, @name=#<Migel::Util::Multilingual:0x7f9d138528e0 @canonical={:de=>"APPLIKATIONSHILFEN"}, @synonyms=[]>, @odba_observers=[], @code="03">
...

Result

migel> Migel::Model::Group.find_by_code('01').class
-> Migel::Model::Group
migel> Migel::Model::Group.find_by_code('01').name
-> ABSAUGGERAETE
migel> Migel::Model::Group.find_by_code('03').class
-> Migel::Model::Group
migel> Migel::Model::Group.find_by_code('03').name.de
-> APPLIKATIONSHILFEN
migel> Migel::Model::Group.find_by_code('01').limitation_text
-> Gerätereparaturen beim Kaufsystem: Bei sorgfältigem Gebrauch ohne Selbstverschuldung, Vergütung nach Aufwand nur nach vorgängiger Kostengutsprache durch den Krankenversicherer.
migel> Migel::Model::Group.find_by_code('03').limitation_text.de
-> Gerätereparaturen beim Kaufsystem: Bei sorgfältigem Gebrauch ohne Selbstverschuldung, Vergütung nach Aufwand nur nach vorgängiger Kostengutsprache durch den Krankenversicherer.

Migel::Model::Subroup

• lib/migel/importer.rb

    def update_subgroup(id, group, row, language)
      subgroupcd = id.at(1)
      subgroup = group.subgroups.find{|sg| sg.code == subgroupcd} || begin
        sg = Migel::Model::Subgroup.new(subgroupcd)
        group.subgroups.push sg
        group.save
        sg
      end
      subgroup.name.send(language.to_s + '=', row.at(6).to_s)

      if text = row.at(7).to_s and !text.empty?
        p text
        subgroup.limitation_text.send(language.to_s + '=', text)
      end
      subgroup.save
      subgroup
    end

• lib/migel/model/subgroup.rb

#!/usr/bin/env ruby
# encoding: utf-8
# Migel::Model::Subgroup -- migel -- 18.08.2011 -- mhatakeyama@ywesee.com

module Migel
  module Model
    class Subgroup < Migel::ModelSuper
      belongs_to :group
      has_many :migelids, on_delete(:cascade), on_save(:cascade)
      attr_reader :code
      alias :pointer_descr :code
      multilingual :limitation_text
      multilingual :name
      def initialize(code)
        @code = code
      end
    end
  end
end

Result

migel> Migel::Importer.new.update('data/csv/migel_de_sample.csv','de')
-> Array

migel> Migel::Model::Group.find_by_code('03').subgroups[2].code
-> 03
migel> Migel::Model::Group.find_by_code('03').subgroups[2].limitation_text
-> Limitation: Zur Zytostatika-, Antibiotika-, Schmerz-, Chelatbildner-, Parkinsontherapie sowie für die parenterale Ernährung.

Migel::Model::Migelid

• lib/migel/model/migelid.rb

#!/usr/bin/env ruby
# encoding: utf-8
# Migel::Model::Migelid -- migel -- 18.08.2011 -- mhatakeyama@ywesee.com

#require 'util/language'
#require 'model/text'
#require 'model/feedback_observer'
#require 'util/searchterms'
#require 'model/migel/item'

module Migel
  module Model
    class Migelid < Migel::ModelSuper
      belongs_to :subgroup
      has_many :products, on_delete(:cascade), on_save(:cascade)
      has_many :accessories
      has_many :migelids
=begin
      include SimpleLanguage
      include FeedbackObserver
      ODBA_SERIALIZABLE = ['@descriptions']
=end
      #attr_reader :code, :accessories, :products, :product_text, :items
      #attr_reader :code, :product_text
      #attr_accessor :subgroup, :limitation, :price, :type, :date, 
      attr_accessor :limitation, :price, :type, :date, :qty, :unit
      attr_reader  :code
      alias :pointer_descr :code
      multilingual :limitation_text
      multilingual :migelid_text
      multilingual :name
      def initialize(code)
        @code = code
      end
      def update_multilingual(data, language)
        data.keys.each do |key|
          self.send(key).send(language.to_s + '=', data[key])
        end
      end
      def add_accessory(acc)
        accessories.push(acc)
      end
      def add_migelid(mi)
        if mi
          unless(migelids.include?(mi))
            mi.add_accessory(self)
            migelids.push(mi)
          end
        end
        mi
      end

• lib/migel/importer.rb

    def update_migelid(id,  subgroup, row, language)

      # take data from csv
      migelidcd = id[2,3].join(".")
      name = row.at(12).to_s
      migelid_text = row.at(15).gsub(/[ \t]+/u, " ")
      migelid_text.tr!("\v", "\n")
      limitation_text = if(idx = migelid_text.index(/Limitation|Limitazione/u))
                          migelid_text.slice!(idx..-1).strip
                        else
                          ''
                        end
      if(name.to_s.strip.empty?)
        name = migelid_text.slice!(/^[^\n]+/u)
      end
      migelid_text.strip!
      type = SALE_TYPES[id.at(4)]
      price = ((row.at(18).to_s[/\d[\d.]*/u].to_f) * 100).round
      date = date_object(row.at(20))
      limitation = (row.at(14) == 'L')
      qty = row.at(16).to_i
      unit = row.at(17).to_s

      # save instance
      migelid = subgroup.migelids.find{|mi| mi.code == migelidcd} || begin
        mi = Migel::Model::Migelid.new(migelidcd)
        subgroup.migelids.push mi
        subgroup.save
        mi
      end
      migelid.subgroup = subgroup
      migelid.save

      multilingual_data = {
        :name            => name,
        :migelid_text    => (migelid_text unless migelid_text.empty?),
        :limitation_text => (limitation_text unless limitation_text.empty?)
      }
      migelid.update_multilingual(multilingual_data, language)
      migelid.type  = type
      migelid.price = price
      migelid.date  = date
      migelid.limitation = limitation
      migelid.qty = qty if qty > 0
      migelid.unit = unit unless unit.empty?

      if(id[3] != "00")
        1.upto(3) { |num|
          micd =  [id[2], '00', num].join('.')
          if mi = subgroup.migelids.find{|m| m.code == micd}
            migelid.add_migelid(mi)
          end
        }
      end

      migelid.save
      migelid
    end
  end
end

Result

migel> Migel::Importer.new.update('data/csv/migel_de_sample.csv','de')
-> Array
migel> a=Migel::Model::Group.find_by_code('03').subgroups[0].migelids[0].class
-> Migel::Model::Migelid
migel> a=Migel::Model::Group.find_by_code('03').subgroups[0].name
-> Applikationshilfen für die künstliche Ernährung
migel> a=Migel::Model::Group.find_by_code('03').subgroups[0].migelids[0].subgroup.name
-> Applikationshilfen für die künstliche Ernährung
migel> a=Migel::Model::Group.find_by_code('03').subgroups[0].migelids[0].limitation
-> false
migel> a=Migel::Model::Group.find_by_code('03').subgroups[0].migelids[0].limitation_text
-> 
migel> a=Migel::Model::Group.find_by_code('03').subgroups[0].migelids[0].migelid_text
-> 
migel> a=Migel::Model::Group.find_by_code('03').subgroups[0].migelids[0].qty
-> 1

Migel::Model::Product

• lib/migel/model/product.rb

#!/usr/bin/env ruby
# encondig: utf-8
# Migel::Model::Product -- migel -- 18.08.2011 -- mhatakeyama@ywesee.com

module Migel
  module Model
    class Product < Migel::ModelSuper
      belongs_to :product, delegates(:price, :unit, :migel_code)
      alias :pointer_descr :migel_code
      attr_accessor :ean_code, :pharmacode, :article_name, :companyname, :companyean, :ppha, :ppub, :factor, :pzr, :size, :status, :datetime, :stdate, :language
      multilingual :article_name
    end
  end
end

Next

• Download from online
• Make a plugin to access to swissindex_nonpharmad
• lib/migel/util/swissindex.rb

#!/usr/bin/env ruby
# enconding: utf-8
# Migel::Util::Swissindex -- migel -- 18.08.2011 -- mhatakeyama@ywesee.com

require 'sbsm/drbserver'

module ODDB
  module Swissindex
    class SwissindexNonpharma; end
  end
end

module Migel
  module Util
    module Swissindex
      SWISSINDEX_NONPHARMA_URI = 'druby://localhost:50002'
      SWISSINDEX_NONPHARMA_SERVER = DRbObject.new(nil, SWISSINDEX_NONPHARMA_URI)


class << self
  def search_item(pharmacode)
    item = {}
    SWISSINDEX_NONPHARMA_SERVER.session(ODDB::Swissindex::SwissindexNonpharma) do |swissindex|
      item = swissindex.search_item_with_swissindex_migel(pharmacode)
    end
    item
  end
end


    end # Swissindex
  end # Util
end # Migel

Run

• bin/migeld
• oddb.org/ext/swissindex/bin/swissindex_nonpharmad

Result

migel> Migel::Util::Swissindex.search_item('2355664')[:article_name]
-> MANUTRAIN Handgelenkband links Gr 6 19-20cm titan

Note

• This is a client to access to the swissindex_nonpharmad (oddb.org) server

Next

• Import product data from online
• lib/migel/importer.rb

    def update_products_by_migel_code(migel_code, lang = 'de')
      if migelid = Migel::Model::Migelid.find_by_migel_code(migel_code)
        migel_code = migelid.migel_code.split('.').to_s
        if table = Migel::Util::Swissindex.search_migel_table(migel_code)
          table.each do |record|
            if record[:pharmacode] and record[:article_name]
              update_product(migelid, record)
            end
          end
        end
      end
    end
    def update_product(migelid, record)
    #  pointer = product.pointer + [:item, record[:pharmacode]]
      product = migelid.products.find{|i| i.pharmacode == record[:pharmacode]} || begin
        i = Migel::Model::Product.new(record[:pharmacode])
        migelid.products.push i
        migelid.save
        i
      end
      product.migelid = migelid
      product.save

      product.ean_code     = record[:ean_code]
      product.article_name = record[:article_name]
      product.companyname  = record[:companyname]
      product.companyean   = record[:companyean]
      product.ppha         = record[:ppha]
      product.ppub         = record[:ppub]
      product.factor       = record[:factor]
      product.pzr          = record[:pzr]
      product.size         = record[:size]
      product.status       = record[:status]
      product.datetime     = record[:datetime]
      product.stdate       = record[:stdate]
      product.language     = record[:language]

      product.save
      product
    end
  end
end

Run

• reset db

masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel
masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel

• bin/migeld
• oddb.org/ext/swissindex/bin/swissindex_nonpharmad
• import migel_de.csv

migel> Migel::Importer.new.update('data/csv/migel_de_sample.csv','de')
-> Array

• import products

migel> Migel::Importer.new.update_products_by_migel_code('03.02.01.00.2')
-> Array

Result (Check)

migel> Migel::Model::Migelid.find_by_migel_code('03.02.01.00.2').products.length
-> 94
migel> Migel::Model::Migelid.find_by_migel_code('03.02.01.00.2').products[0].pharmacode
-> 4438897
migel> Migel::Model::Migelid.find_by_migel_code('03.02.01.00.2').products[0].article_name
-> ACCU CHEK COMBO Service Pack normal
migel> Migel::Model::Product.find_by_pharmacode('4438897').article_name
-> ACCU CHEK COMBO Service Pack normal
migel> Migel::Model::Migelid.find_by_migel_code('03.02.01.00.2').products[0].uid
-> 142
migel> Migel::Model::Product.find_by_pharmacode('4438897').uid
-> 142

Note

• The 'pharmacode' of 'find_by_pharmacode' method is set as 'odba_index' in lib/migel/persistence/odba/model/product.rb
• 'find_by_XXX' is like find method of Array, which returns the first element that is matched to the condition
• 'search_by_XXX' returns Arrany in which the elements are matched to the condition

Next

• Import all the Group, Subgroup, Migelid from csv file and Product from online
• src/migel/importer.rb

    def update_all_products(lang = 'de')
      Migel::Model::Migelid.all.each do |migelid|
        update_products_by_migel_code(migelid.migel_code)
      end
    end

Result

migel> Migel::Model::Migelid.all.length
-> 44
migel> Migel::Model::Product.all.length
-> 94

migel> Migel::Importer.new.update_all_products
-> Array
(after update_all_products)

migel> Migel::Model::Migelid.all.length
-> 44
migel> Migel::Model::Product.all.length
-> 644

Final import (only 'de')

Run

• oddb.org/ext/swissindex/bin/swissindex_nonpharmad
• reset database

masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel
masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel

• bin/migeld
• import migel data(group, subgroup, migelid) from csv file

migel> Migel::Importer.new.update('data/csv/migel_de.csv','de')

• import product data from online

migel> Migel::Importer.new.update_all_products true

Log

...
15 / 571        Estimate total: 1.57 [h] It will be done in: 1.53 [h]
16 / 571        Estimate total: 1.52 [h] It will be done in: 1.48 [h]
17 / 571        Estimate total: 1.44 [h] It will be done in: 1.39 [h]
18 / 571        Estimate total: 1.37 [h] It will be done in: 1.32 [h]
...

Note

• I expect that it takes about 12 hours
• Lets see the result tomorrow

Wrapper class of migel data in oddb.org

• A plugin is necessary to access to the migeld server from oddb.org