20110816-debug-migel-oddb_org

<< | Index | >>

1. Confirm the result of update_migel_nonpharma
2. Debug migel_code view
3. Debug updater exporter errors
4. Update migel data again
5. Debug search result
6. Set crontab for update_migel_nonpharma
7. Count migel items that have ean_code

1. Implement search function by artcle name

Goal/Estimate/Evaluation

• Debug Migel / 100% / 100%

Milestones

• Update migel data if article name is nil
• Search by article name
• Search by company name
• Comparison with cost
• Compress exported files suspend
• Import and show fr migel data suspend

Summary

• http://masay.ch/2011/08/16/mainly-debug-migel-function-in-oddb-org/

Commits

• Fixed email address, Migel::Product view, swissindex bug. Added jobs/import_mige_nonpharma. Created new methods to update migel data in MiGeLPlugin. (oddb.org)

ToDo Tomorrow

• Search by article name
• Search by company name
• Comparison with cost

Confirm the result of update_migel_nonpharma

Email ch.ODDB.org Report - Updated Migel Nonpharma - 08/2011

Updated MiGel items: 29943

/var/www/oddb.org/data/csv/swissINDEX_MiGel.csv
swissINDEX_MiGel.csv 4902K

Note

• The number of migel code is 571
• In average, 29943 / 571 = 52.4 items (pharmacode) per product (migelcode)
• It took about 12 hours to run (update + export)

Debug migel_code view

Problem

• Migel code title is not shown when the migel code is not clickable.

• src/view/migel/product.rb#migel_code

   def migel_code(model)
    if items = model.items and !items.empty?
      link = PointerLink.new(:to_s, model, @session, self)
      link.value = model.migel_code
      link.href = @lookandfeel._event_url(:migel_search, {:migel_code => model.migel_code.gsub(/\./, '')})
      link
    else
      value = HtmlGrid::Value.new(:to_s, model, @session)
      value.value = model.migel_code
      value
    end
   end

Result

Debug updater exporter errors

Email

• Fwd: ch.ODDB.org Report - Error: swiss - 08/2011
• Fwd: ch.ODDB.org Report - Error Export: oddb.yaml - 08/2011

Note

• Updater.new(self).updater_swissmedic runs locally without error (though the source code and data are totally same with the online server)
• Exporter.new(self).export_yaml also looks running without error

Consideration

• update_migel_nonpharma method took about 12 hours, and the other updater and exporter methods also ran at the same time called by import_daily job
• This might make a collision of two objects on ODBA cache
• We should review the ODBA update mechanism in the future

Update migel data again

Problem

• Some item data is missing
• Probably because the server does not response over 3 times

Solution

• all the migel data not having article name should be update again

Next

• Implement a method to call item data not by 'migel code' but 'pharmacode' in ext/swissindex/src/swissindex.rb

Experiment

• ext/swissinex/src/swissindex.rb

  def merge_swissindex_migel(swissindex_item, migel_line)
    # Swissindex data
    swissindex = swissindex_item.collect do |key, value|
      case key
      when :gtin
        [:ean_code, value]
      when :dt
        [:datetime, value]
      when :lang
        [:language, value]
      when :dscr
        [:article_name, value]
      when :addscr
        [:size, value]
      when :comp
        [:compnayname, value[:name], :companyean, value[:gln]]
      else
        [key, value]
      end
    end
    swissindex = Hash[*swissindex.flatten]

    # Migel data
    pharmacode, article_name, companyname, ppha, ppub, factor, pzr = *migel_line
    migel = {
      :pharmacode   => pharmacode,
      :article_name => article_name,
      :companyname  => companyname,
      :ppha         => ppha,
      :ppub         => ppub,
      :factor       => factor,
      :pzr          => pzr,
    }
    migel.update swissindex
  end
  def search_item_with_swissindex_migel(pharmacode)
    migel_line = search_migel(pharmacode)
    if swissindex_item = search_item(pharmacode)
      merge_swissindex_migel(swissindex_item, migel_line)
    else
      merge_swissindex_migel({}, migel_line)
    end
  end

• src/plugin/swissindex.rb#search_migel_item

    def search_item(pharmacode)
      item = {}
      SWISSINDEX_NONPHARMA_SERVER.session(ODDB::Swissindex::SwissindexNonpharma) do |swissindex|
        item = swissindex.search_item_with_swissindex_migel(pharmacode)
      end
      item
    end

• src/plugin/migel.rb#update_migel_nonpharma_again_by_pharma

    # This is just to update items which do not have article_name
    def update_items_again_by_pharma(time_estimate = false)
      total = @app.migel_count
      start_time = Time.now
      @count_updated_item = 0
      @output_file = nil
      plugin = ODDB::SwissindexNonpharmaPlugin.new(@app)
      @app.migel_products.each_with_index do |product, count|
        if items = product.items
          items.each do |pharmacode, item|
            unless item.article_name
              item_record = plugin.search_item(pharmacode)
              update_item(product, item_record)
              @count_updated_item += 1
            end
          end
        end
        puts estimate_time(start_time, total, count + 1) if time_estimate
      end
      return true
    end

Run

• bin/admin

ch.oddb> ODDB::MiGeLPlugin.new(self).update_items_again_by_pharma

Result

Debug search result

Problem

• An error comes regarding Migel::Item class in the search result

Note

• Some Package#@narcotics are Migel::Item instances, this should be Array

Cause

• It may be a cause that update_migel_nonpharma and the other updaters ran last night.

Set crontab for update_migel_nonpharma

Directly rewrite /etc/crontab

1 23 2 * *       user  /var/www/oddb.org/jobs/import_migel_nonpharma

• jobs/import_migel_nonpharma

#!/usr/bin/env ruby
# must be scheduled in crontab to run as the same user as oddb

$: << File.expand_path('../src', File.dirname(__FILE__))
$: << File.expand_path('..', File.dirname(__FILE__))

require 'util/job'
require 'util/updater'

module ODDB
  module Util
    Job.run do |system|
      Updater.new(system).update_migel_nonpharma
    end
  end
end

Count migel items that have ean_code

Run

ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.length end}; c
-> 29943
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(ODDB::Migel::Item) }.length end}; c
-> 29943
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.ean_code}.length end}; c
-> undefined method `ean_code' for #<Array:0x7f010487f440>
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| !it.is_a?(Array) && !it.is_a?(Part) && !it.is_a?(Package) && !it.is_a?(ActiveAgent) && !it.is_a?(Composition) && !it.is_a?(Hash) && !it.is_a?(Sequence) && it.ean_code }.length end}; c
-> 24467

Note

• Something is wrong
• All the instances should be of Migel::Item
• Some instances are not Migel::Item

More

ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Array)  }.length end}; c
-> 15
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Part)}.length end}; c
-> 9
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Package)}.length end}; c
-> 3
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(ActiveAgent)}.length end}; c
-> 4
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Composition)}.length end}; c
-> 3
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Hash)}.length end}; c
-> 4
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Sequence)}.length end}; c
-> 3
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Array) or it.is_a?(Part) or it.is_a?(Package) or it.is_a?(ActiveAgent) or it.is_a?(Composition) or it.is_a?(Hash) or it.is_a?(Sequence)}.length end}; c
-> 41
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(ODDB::Migel::Item)}.length end}; c
-> 29943
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| (it.is_a?(Array) or it.is_a?(Part) or it.is_a?(Package) or it.is_a?(ActiveAgent) or it.is_a?(Composition) or it.is_a?(Hash) or it.is_a?(Sequence)) and it.is_a?(ODDB::Migel::Item)}.length end}; c
-> 41
ch.oddb> c=0; migel_products.each{|pr| if items=pr.items then c+=items.values.select{|it| it.is_a?(Array) and it.is_a?(ODDB::Migel::Item)}.length end}; c
-> 15

Consideration

• What does this mean?

More

ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a.class
-> ODDB::Migel::Item
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a.article_name
-> undefined method `article_name' for #<Array:0x7f010473de38>
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a.length
-> 3
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a[0]
-> Aliskirenum 300 mg
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a[1]
-> Amlodipinum 10 mg
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a[2]
-> Hydrochlorothiazidum 12.5 mg
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a[2].class
-> ODDB::ActiveAgent
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a[1].class
-> ODDB::ActiveAgent
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a[0].class
-> ODDB::ActiveAgent
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a[0].pointer
-> :!registration,61678!sequence,03!composition,27218368!active_agent,Aliskirenum.

ch.oddb> registration('61678').sequence('03').compositions[0].active_agents[0].pointer
-> :!registration,61678!sequence,03!composition,27218368!active_agent,Aliskirenum.

Consideration

• Anyway, it is fact that some Migel::Item instances are over written with some other instance
• And in particular, this is actually a bug, in odba or bin/admin or somewhere

ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a.class
-> ODDB::Migel::Item
ch.oddb> a=nil; migel_products.each{|pr| if items=pr.items then items.values.each{|it| if it.is_a?(Array) then a=it end} end}; a.article_name
-> undefined method `article_name' for #<Array:0x7f010473de38>

Conclusion

• We should execute Updater.new(self).update_migel_nonpharma again and update all the migel item data
• Then let's check all the item class if they are Migel::Item class or not

Commit

• Fixed email address, Migel::Product view, swissindex bug. Added jobs/import_mige_nonpharma. Created new methods to update migel data in MiGeLPlugin. (oddb.org)

Implement search function by artcle name

Experiment

• Access: http://oddb.masa.org/de/gcc/search/zone/migel/search_query/Astra#best_result

Result

• Nothing comes

Next

• Check src/state/global.rb#search method
• src/state/global.rb#search

      def search
        zone = @session.zone
...
          case zone
...
          when :migel
            result = @session.search_migel_products(query)
            State::Migel::Result.new(@session, result)

Note

• The search function only searches migel products

Next

• Create a new method to search migel items by article name
• Create new state and view for the search result by article name