20110913-update-importer-testcases-migel

<< | Index | >>

1. Testcases migel
2. 'delete' method
3. Update importer with 'delete' method

Goal/Estimate/Evaluation:

• Testcases migel / 100% / 98%
• Update migel importer / 100% / 90%

Milestones

1. testcases migel
2. 'delete' method
3. 'could not sort' warning -> this is caused by probably to_s method
4. put 'delete' method into the import process
5. Testcases
6. Importer job
7. Debug RangeError

Summary

• http://masaywesee.wordpress.com/2011/09/13/update-importer-testcases-in-migel/

ToDo

• spec for lib/migel/server.rb#init_fullindex_text_tables
• require 'migel/util/m10l_document' problem in multilingual.rb
• require 'oddb/util/m10l_document' problem in model_super.rb

Testcases migel

1. util
   1. oddbapp.rb (coverage: 87.79%)

Note

• yusd and migeld must run in order to execute tesetcases
• The testcases for the migel process is finished
• But there are some method which are not tested in oddbapp.rb

suspend

'delete' method

Experiment

migel> Migel::Model::Product.find_by_pharmacode('2126923').article_name
-> CONTAM Vaginaltampon 26mm regular
migel> Migel::Model::Product.find_by_pharmacode('2126923').delete
-> CONTAM Vaginaltampon 26mm regular
migel> Migel::Model::Product.find_by_pharmacode('2126923').article_name
-> undefined method `article_name' for nil:NilClass
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').delete
-> Stück
migel> Migel::Model::Product.find_by_pharmacode('2126975').article_name
-> CONTAM Vaginaltampon 33mm extra 5 Stk
migel> Migel::Model::Product.find_by_pharmacode('2126975').migelid.class
-> NilClass

Note

• 'delete' method takes long time (about 1 minute)
• This is probably because of updating of fullindex table

Problem

• It seems that some product instances remain even though a Migelid instance is deleted by 'delete' method
• It seems that 'on_delete(:cascade)' method does not work

Experiment

• lib/migel/model/migelid.rb

      def delete
        products.each do |product|
          product.delete
        end
        super
      end

Restore

masa@masa ~/ywesee/migel_dev $ sudo -u postgres dropdb migel; sudo -u postgres createdb -E UTF8 -T template0 migel
masa@masa ~/ywesee/migel_dev $ cat migel_database_with_fullindex_tables_backup.20110913.sql |psql -U postgres migel

Note

• This sql includes fullindex text tables

Run

• bin/migeld
• bin/admin

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').delete

Result

migel> Migel::Model::Product.find_by_pharmacode('2998899')
-> CONTRELLE Starterpack G1 G2 G3 ass 3 Stk
migel> Migel::Model::Product.find_by_pharmacode('2998899').migelid.class
-> NilClass
migel> Migel::Model::Product.find_by_pharmacode('2126981')
-> CONTAM Vaginaltampon Probierset ass 3 Stk
migel> Migel::Model::Product.find_by_pharmacode('2126981').migelid.class
-> NilClass

Experiment

• lib/migel/model/migelid.rb

       def delete
        products.each do |product|
          p product.odba_id
          p product.pharmacode
          product.delete
        end
        p products.length
        #super
      end

Run

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').delete

Log

9919
"2126917"
9921
"2126975"
9923
"2760086"
9925
"2760100"
4

• bin/admin

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.map{|pr| pr.pharmacode.to_s}.join(",")
-> 2126923,2126981,2760092,2998899

Note

• 'Migelid#products.each' looks strange

Experiment

• Restore
• bin/admin

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.map{|pr| pr.pharmacode.to_s}.join(",")
-> 2126917,2126923,2126975,2126981,2760086,2760092,2760100,2998899

Note

• Maybe the 'delete' should not be used in the 'each' iteration

Experiment

• lib/migel/model/migelid.rb

      def delete
        while product = products.shift
          p product.odba_id
          p product.pharmacode
          product.delete
        end
        p products.length
        #super
      end

Run

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.map{|pr| pr.pharmacode.to_s}.join(",")
-> 2126917,2126923,2126975,2126981,2760086,2760092,2760100,2998899
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').delete
->
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.map{|pr| pr.pharmacode.to_s}.join(",")
-> 
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.length
-> 0

Log

9919
"2126917"
9920
"2126923"
9921
"2126975"
9922
"2126981"
9923
"2760086"
9924
"2760092"
9925
"2760100"
9926
"2998899"
0

Note

• Success

IMPORTANT

• This 'delete' bug is also hidden in de.oddb.org

Experiment

• lib/migel/model_super.rb

    def delete
      while predicate = self.class.predicates.shift
        predicate.execute(:delete, self)
      end
      self
    end

Run

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.map{|pr| pr.pharmacode.to_s}.join(",")
-> 2126917,2126923,2126975,2126981,2760086,2760092,2760100,2998899
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').delete

Result

• This does not work

Experiment

• lib/migel/model_super.rb

      def cascade(action, next_level)
        if(next_level.is_a?(Array))
          if action == :delete
            while element = next_level.shift
              cascade(action, element)
            end
          else
            next_level.each { |element|
              cascade(action, element)
            }
          end
        else
          next_level.send(action) if(next_level.respond_to?(action))
        end
      end

Run

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.map{|pr| pr.pharmacode.to_s}.join(",")
-> 2126917,2126923,2126975,2126981,2760086,2760092,2760100,2998899
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').delete
-> Stück
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').products.map{|pr| pr.pharmacode.to_s}.join(",")
-> undefined method `products' for nil:NilClass
migel> 

Result

• Success

Note

• The testcases for model_super.rb are not implemented
• 'delete' method, in particular for Group and Subgroup, takes long time

Update importer with 'delete' method

Algorithm (Design)

1. keep all the current migel_code (Group, Subgroup, Migelid) and pharmacode (Product) and save them as delete_code_list
2. keep the updated code during the updating process, and delete them from the delete_code_list
3. After the updating, delete the instances of the delete_code_list
4. rebuild_fulltext_index_tables

Experiment

• lib/migel/util/importer.rb

  def update(path, language)
    # update Group, Subgroup, Migelid data from a csv file
    CSV.readlines(path)[1..-1].each do |row|
      id = row.at(13).to_s.split('.')
      if(id.empty?)
        id = row.at(4).to_s.split('.')
      else
        id[-1].replace(id[-1][0,1])
      end
      unless id.empty?
        group = update_group(id, row, language)
        subgroup = update_subgroup(id, group, row, language)
        migel_code_list.delete(group.migel_code)
        migel_code_list.delete(subgroup.migel_code)
        if(id.size > 2)
          migelid = update_migelid(id, subgroup, row, language)
          migel_code_list.delete(migelid.migel_code)
        end
      end
    end

    # delete not updated list
    migel_code_list.each do |migel_code|
      case migel_code.length
      when 2
        Migel::Model::Group.find_by_migel_code(migel_code).delete
      when 5
        Migel::Model::Subgroup.find_by_migel_code(migel_code).delete
      else
        Migel::Model::Migelid.find_by_migel_code(migel_code).delete
      end
    end
  end

  def code_list(index_table_name, output_filename = nil)
    list = ODBA.cache.index_keys(index_table_name)
    if output_filename
      File.open(output_filename, 'w') do |out|
        out.print list.join("\n"), "\n"
      end
    end
    list
  end
  def migel_code_list(output_filename = nil)
    @migel_code_list ||= begin
      index_table_name = 'migel_model_migelid_migel_code'
      code_list(index_table_name, output_filename)
    end
  end

  def import_all_products_from_csv(file_name = 'migel_products_de.csv', lang = 'de', estimate = false)
    ..
        update_product(migelid, record, lang)
        pharmacode_list.delete(line[1])
      end
      puts "updating: " + estimate_time(start_time, total, count) if estimate
    end
    ..
    # delete process
    pharmacode_list.each do |pharmacode|
      Migel::Model::Product.find_by_pharmacode(pharmacode).delete
    end

end

Restore

masa@masa ~/ywesee/migel_dev $ sudo -u postgres dropdb migel; sudo -u postgres createdb -E UTF8 -T template0 migel
masa@masa ~/ywesee/migel_dev $ bin/migeld

Run

• bin/migeld
• bin/admin

migel> Migel::Util::Importer.new.update('data/csv/migel_de.csv','de')
->
migel> Migel::Util::Importer.new.update('data/csv/migel_fr.csv','fr')
->
migel> init_fulltext_index_tables
-> 
migel> Migel::Util::Importer.new.update('test.csv','de')
-> 15.30.50.00.1

Result

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').class
-> NilClass

Note

• Looks good

Restore

masa@masa ~/ywesee/migel_dev $ sudo -u postgres dropdb migel; sudo -u postgres createdb -E UTF8 -T template0 migel
masa@masa ~/ywesee/migel_dev $ cat migel_database_backup.20110913.sql |psql -U postgres migel
masa@masa ~/ywesee/migel_dev $ bin/migeld

Run

• bin/migeld
• bin/admin

migel> Migel::Util::Importer.new.import_all_products_from_csv('test.csv', 'de', true)

Result

migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').class
-> Migel::Model::Migelid
migel> Migel::Model::Migelid.find_by_migel_code('15.30.50.00.1').name.de
-> Wegwerf-Scheidenpessar

Note

• failed
• The data is not deleted