20110829-import-fr_data-migel-drb-client-oddb_org

<< | Index | >>

1. Import migel data in both 'de' and 'fr'
   1. Debug Migelid import
   2. Check Product import
   3. Change Size class from String to Multilingual
2. Testcases swissindex
3. Testcases migel suspend

1. Keep Original/Generic type

Goal/Estimate/Evaluation:

• Migel drb client in oddb.org / 100% / 95%

Milestones

1. import fr data
   1. check migelid fr import
   2. check product fr import
2. Testcase swissindex
   1. test_search_migel_table
   2. test_search_item_with_swissindex_migel
3. Testcase migel suspend
4. Final import

Summary

• http://masaywesee.wordpress.com/2011/08/29/import-french-data-in-migel-drb-server/

Commits

• Updated search methods for both DE and FR (oddb.org)
• Fix search_migel_table method in french search (oddb.org)
• Updated swissindex testcases (oddb.org)

Keep in Mind

• alias article_name and company_name Migel::Model::Product

ToDo

• Refactor Group, Subgroup, Migelid because some code in the classes are totally same
• Check Group and Subgroup de and fr view
• Testcases oddb.org, swissindex.rb, migel

Import migel data in both 'de' and 'fr'

Log in the last Friday

migel> Migel::Importer.new.update_all_products true
-> /home/masa/ywesee/migel/lib/migel/importer.rb:200:in `search_migel_table': wrong number of arguments (2 for 1)

Commit

• Updated search methods for both DE and FR (oddb.org)

Experiment

• Restore database

masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel
masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel
masa@masa ~/ywesee/migel $ bin/migeld

• Load migel_de.csv

migel> Migel::Importer.new.update('data/csv/migel_de_test.csv','de')

• Load migel_fr.csv

migel> Migel::Importer.new.update('data/csv/migel_fr_test.csv','fr')

Problem

• alphabetical search failed (German article_names disappear but search function finds them)
• nil error comes when the alphabetical search

NoMethodError
undefined method `<=>' for nil:NilClass
(druby://localhost:33000) /home/masa/ywesee/migel/lib/migel/util/multilingual.rb:31:in `sort'
(druby://localhost:33000) /home/masa/ywesee/migel/lib/migel/util/multilingual.rb:31:in `to_s'
/home/masa/ywesee/oddb.org/src/view/migel/result.rb:73:in `limitation_text'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:66:in `send'
/usr/lib64/ruby/site_ruby/1.8/htmlgrid/composite.rb:66:in `create'

Log on console

WARNING:  nonstandard use of \' in a string literal
LINE 2:           VALUES (39, 'catetere monouso con lubrificante / c...
                              ^
HINT:  Use '' to write quotes in strings, or use the escape string syntax (E'...').
WARNING:  nonstandard use of \' in a string literal
LINE 2:           VALUES (41, 'catetere monouso con lubrificante / c...
                              ^
HINT:  Use '' to write quotes in strings, or use the escape string syntax (E'...').
WARNING:  nonstandard use of \' in a string literal
LINE 2:           VALUES (103, 'apparecchi per la terapia dell\'enur...
                               ^
HINT:  Use '' to write quotes in strings, or use the escape string syntax (E'...').

Note

• after loading fr data, the fr data is shown in 'de' view, and 'fr' view shows nothing other than migel code
  1. Debug Migelid import

Experiment

• lib/migel/importer.rb#update_migelid

      multilingual_data = {
        :name            => name,
        :migelid_text    => migelid_text,
        :limitation_text => limitation_text,
        :unit => unit,
      }

Result

• Error does not happen anymore, but stil alphabetical search is not correct

Experiment

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').name.fr
-> Pessaire jetable
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').name.de
-> Wegwerf-Scheidenpessar

migel> Migel::Model::Migelid.find_by_name('Pessaire jetable').migel_code
-> 15.30.50.00.1
migel> Migel::Model::Migelid.find_by_name('Wegwerf-Scheidenpessar').class

Note

• Probably the cause is 'find_by' and 'search_by' method

Question

• Why is the German name not found by 'find_by_name' method?

Experiment

• lib/migel/persistence/odba/model/migelid.rb

      odba_index :name_de, 'name.de'
      odba_index :name_fr, 'name.fr'

• src/util/oddbapp.rb

    def search_migel_alphabetical(query, lang)
      #MIGEL_SERVER.migelid.search_by_name(query)
      search_method = 'search_by_name_' + lang.downcase.to_s
      MIGEL_SERVER.migelid.send(search_method, query)
    end

Result

migel> Migel::Model::Migelid.find_by_name_fr('Pessaire jetable').migel_code
-> 15.30.50.00.1
migel> Migel::Model::Migelid.find_by_name_de('Wegwerf-Scheidenpessar').migel_code
-> 15.30.50.00.1

• Good

Next

• The other names of Subgroup and Group should be updated, too.
• lib/migel/persistence/odba/model/group.rb, subgroup.rb

      odba_index :name_de, 'name.de'
      odba_index :name_fr, 'name.fr'

• src/util/oddbapp.rb

    def search_migel_products(query, lang)
      migel_code = if query =~ /(\d){9}/
                     query.split(/(\d\d)/).select{|x| !x.empty?}.join('.')
                   elsif query =~ /(\d\d\.){4}\d/
                     query
                   end
      if migel_code
         MIGEL_SERVER.migelid.search_by_migel_code(migel_code)
      else
        search_method = 'search_by_name_' + lang.downcase.to_s
        #if group = MIGEL_SERVER.group.search_by_name(query).first
        if group = MIGEL_SERVER.group.send(search_method, query).first
          result = group.subgroups.collect{|sg| sg.migelids}.flatten
        #elsif subgroup = MIGEL_SERVER.subgroup.search_by_name(query).first
        elsif subgroup = MIGEL_SERVER.subgroup.send(search_method, query).first
          subgroup.migelids
        else
          #MIGEL_SERVER.migelid.search_by_name(query)
          MIGEL_SERVER.migelid.send(search_method, query)
        end
      end
    end

Restore database

masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel
masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel
masa@masa ~/ywesee/migel $ bin/migeld

migel> Migel::Importer.new.update('data/csv/migel_de_one.csv','de')
-> 15.INKONTINENZHILFEN15.30Pessare15.30.50.00.1Wegwerf-Scheidenpessar1Stück3.60 01.01.2002
migel> Migel::Importer.new.update('data/csv/migel_fr_one.csv','fr')
-> 15.AIDES POUR L’INCONTINENCE15.30Pessaires15.30.50.00.1Pessaire jetable1pièce3.60 01.01.2002

Run

• bin/oddbd
• bin/migeld

Result

• alphabetical search by both de and fr: Good
• group, subgroup, migelid name search by both de and fr: Good

Next

• I should check limitation text too. Looks working
• Product search should be updated too.

Check Product import

• lib/migel/persistence/odba/model/product.rb

module Migel
  module Model
    class Product < ModelSuper
      odba_index :pharmacode
      odba_index :ean_code
      #odba_index :article_name
      odba_index :name_de, 'name.de'
      odba_index :name_fr, 'name.fr'
      #odba_index :companyname
      odba_index :company_name_de, 'company_name.de'
      odba_index :company_name_fr, 'company_name.fr'
    end
  end
end

• src/state/global.rb#search

            #elsif result = @session.app.search_migel_items(query) and !result.empty?
            elsif result = @session.app.search_migel_items(query, @session.language) and !result.empty?
              State::Migel::Items.new(@session, result)

• src/util/oddbapp.rb#search_migel_items

    def search_migel_items(query, lang)
      search_method_article_name = 'search_by_name_' + lang.downcase.to_s
      search_method_company_name = 'search_by_company_name_' + lang.downcase.to_s
      #MIGEL_SERVER.product.search_by_article_name(query) + MIGEL_SERVER.product.search_by_companyname(query)
      MIGEL_SERVER.product.send(search_method_article_name, query) + MIGEL_SERVER.product.send(search_method_company_name, query)
    end

• lib/migel/importer.rb#update_product

    def update_product(migelid, record, lang = 'de')
      lang.downcase!
...
      #product.article_name = record[:article_name]
      product.send(:article_name).send(lang.to_s + "=", record[:article_name])
      #product.companyname  = record[:companyname]
      product.send(:companyname).send(lang.to_s + "=", record[:companyname])

• ext/swissindex/src/swissindex.rb ( Fix search_migel_table method in french search (oddb.org))

  def search_migel_table(code, query_key = 'Pharmacode', lang = 'DE')
...
          migel_item = if pharmacode = line[0] and pharmacode.match(/\d{7}/) and swissindex_item = search_item(pharmacode, lang)

Restore database

masa@masa ~/ywesee/migel $ sudo -u postgres dropdb migel
masa@masa ~/ywesee/migel $ sudo -u postgres createdb -E UTF8 -T template0 migel
masa@masa ~/ywesee/migel $ bin/migeld

migel> Migel::Importer.new.update('data/csv/migel_de_test.csv','de')
-> Array
migel> Migel::Importer.new.update('data/csv/migel_fr_test.csv','fr')
-> Array
migel> Migel::Importer.new.update_products_by_migel_code('15.30.50.00.1', 'de')
-> Array
migel> Migel::Importer.new.update_products_by_migel_code('15.30.50.00.1', 'fr')
-> Array

Result

• German result

• French result

Note

• 'size' also should be updated as Multilingual class (It is now String)
• The '5 pce' is overwritten by french data above

Change Size class from String to Multilingual

• lib/migel/model/product.rb

      multilingual :size

• lib/migel/importer.rb#update_product

      #product.size         = record[:size]
      product.send(:size).send(lang.to_s + "=", record[:size])

• src/view/migel/items.rb

  def size(model = @model, session = @session)
    if model.size.respond_to?(session.language)
      model.size.send(session.language)
    else
      model.size
    end
  end

Result

• German product

• French product

• Good

Testcases swissindex

• ext/swissindex/test/test_swissindex.rb (Coverage: 100%)

Commit

• Updated swissindex testcases (oddb.org)

Keep Original/Generic type

Task

• Email Overwriting Generikum/SL Flag from BAG
• Make a check box next to 'Original/Generic' pull button in Registration configuration view
• If the check box is checked, the value is not updated

Design

• A new value in Registration class is necessary
• update_bsv should be updated

Note

• Original: Red
• Generic: Green
• None: Black

Experiment

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> registration('61848').generic_type
-> generic
ch.oddb> registration('61848').generic_type
-> original
ch.oddb> registration('61848').generic_type.class
-> Symbol
ch.oddb> registration('61848').generic_type.class
-> NilClass

Note

• 'generic_type' variable keeps the value as Symbol