20110906-update-importer-testcases-migel
<< Masa.20110907-update-importer-testcase-spreadsheet-testcases-migel | 2011 | Masa.20110905-update-degi-flavor-oddb_org-502-entry-xmlconv-debug-importer-testcases-migel >>
Goal/Estimate/Evaluation:
- Update importer migel / 80% / 90%
- Milestones
update importer- testcases migel
- Summary
- ToDo
- Testcases oddb.org, migel
- at the same time, refactoring Group, Subgroup, Migelid because some code in the classes are totally same
- refactor rebuild_fulltext_index_tables
- delete_migelids in lib/migel/util/server.rb
- 'could not sort' warning -> this is caused by probably to_s method
- RangeError 0x3fd07d5c4ee0 is recycled object
Update importer migel
Task
- Divide the import into the following two processes
- Save all the product data into a csv file
- Import the data from the csv file
Experiment
- lib/migel/importer.rb
def save_all_products(lang = 'de', csvfilename = 'migel_products_de.csv', estimate = false, skip = -1)
lang.upcase!
start_time = Time.now
index_table_name = 'migel_model_migelid_migel_code'
migel_codes = ODBA.cache.index_keys('migel_model_migelid_migel_code')
total = migel_codes.length
CSV.open(filename, 'w') do |writer|
migel_codes.each_with_index do |migel_code, count|
if count > skip
get_products_by_migel_code(migel_code, lang).each do |record|
writer << [
migel_code,
record[:pharmacode],
record[:ean_code],
record[:article_name],
record[:companyname],
record[:companyean],
record[:ppha],
record[:ppub],
record[:factor],
record[:pzr],
record[:size],
record[:status],
record[:datetime],
record[:stdate],
record[:language],
]
end
end
puts estimate_time(start_time, total, count+1) if estimate
end
end
end
def get_products_by_migel_code(migel_code, lang = 'de')
lang.upcase!
products = if migelid = Migel::Model::Migelid.find_by_migel_code(migel_code)
migel_code = migelid.migel_code.split('.').to_s
if table = Migel::Util::Swissindex.search_migel_table(migel_code, lang)
table.select{|record| record[:pharmacode] and record[:article_name]}
end
end
products
end
Restore
masa@masa ~/ywesee/migel_dev $ sudo -u postgres dropdb migel_test; sudo -u postgres createdb -E UTF8 -T template0 migel_test masa@masa ~/ywesee/migel_dev $ bin/migeld server_url='druby://localhost:77777' db_name='migel_test'
migel> Migel::Importer.new.update('data/csv/migel_de_test.csv','de')
Run
- bin/migeld
- bin/admin
migel> Migel::Importer.new.save_all_products('de', 'migel_products_de.csv', true)
Log
1 / 37 Estimate total: 6.77 [h] It will be done in: 6.59 [h] 2 / 37 Estimate total: 4.04 [h] It will be done in: 3.82 [h] 3 / 37 Estimate total: 2.96 [h] It will be done in: 2.72 [h] 4 / 37 Estimate total: 2.56 [h] It will be done in: 2.29 [h] 5 / 37 Estimate total: 2.06 [h] It will be done in: 1.78 [h] 6 / 37 Estimate total: 1.96 [h] It will be done in: 1.64 [h] ... 36 / 37 Estimate total: 35.82 [m] It will be done in: 0.97 [m] 37 / 37 Estimate total: 35.12 [m] It will be done in: -0.00 [m]
Result
- Looks good
Next
- Import from a csv file
Experiment
- lib/migel/importer.rb
def import_all_products_from_csv(file_name = 'migel_products_de.csv', lang = 'de', estimate = false)
lang.upcase!
start_time = Time.new
total = File.readlines(file_name).to_a.length
count = 0
CSV.open(file_name, 'r') do |line|
count += 1
migel_code = line[0]
if migelid = Migel::Model::Migelid.find_by_migel_code(migel_code)
record = {
:pharmacode => line[1],
:ean_code => line[2],
:article_name => line[3],
:companyname => line[4],
:companyean => line[5],
:ppha => line[6],
:ppub => line[7],
:factor => line[8],
:pzr => line[9],
:size => line[10],
:status => line[11],
:datetime => line[12],
:stdate => line[13],
:language => line[14],
}
update_product(migelid, record, lang)
end
puts estimate_time(start_time, total, count) if estimate
end
end
Run
- bin/migeld
- bin/admin
migel> Migel::Importer.new.import_all_products_from_csv('migel_products_de.csv','de', true)
Log
... 2356 / 2358 Estimate total: 5.54 [m] It will be done in: 0.00 [m] 2357 / 2358 Estimate total: 5.54 [m] It will be done in: 0.00 [m] 2358 / 2358 Estimate total: 5.54 [m] It will be done in: -0.00 [m]
Result
- No error
Note
- There are some missing products which are not imported
- These are probably skipped because of a server connection failure or there is no entry found on the server
Experiment
- lib/migel/importer.rb
def unimported_migel_code_list(output_filename = nil)
migle_codes = migel_code_list.select do |migel_code|
migelid = Migel::Model::Migelid.find_by_migel_code(migel_code) and migelid.products.empty?
end
if output_filename
open(output_filename, 'w') do |out|
out.print migel_codes.join("\n"), "\n"
end
else
migel_codes
end
end
Run
- bin/migeld
- bin/admin
migel> Migel::Importer.new.unimported_migel_code_list('unimported_migel_code_list.dat')
Result
masa@masa ~/ywesee/migel_dev $ cat unimported_migel_code_list.dat 15.11.02.00.1 15.20.01.02.2 15.30.50.00.1
Run importer (the original migel_de.csv and migel_fr.csv)
Restore
masa@masa ~/ywesee/migel_dev $ sudo -u postgres dropdb migel; sudo -u postgres createdb -E UTF8 -T template0 migel masa@masa ~/ywesee/migel_dev $ bin/migeld
Import
migel> Migel::Importer.new.update('data/csv/migel_de.csv','de')
-> Array
migel> Migel::Importer.new.update('data/csv/migel_fr.csv','fr')
-> Array
migel> Migel::Importer.new.save_all_products('migel_product_de.csv', 'de', true)
migel> Migel::Importer.new.save_all_products('migel_product_fr.csv', 'fr', true)
Result
- Let's see result tomorrow
Memo (rspec, rcov, rake)
masa@masa ~/ywesee/migel_dev $ rake -f spec/suite.rake (in /home/masa/ywesee/migel_dev) rm -r coverage /usr/bin/ruby18 -S rcov -Ispec:lib -x /home/redcap/.rvm/gems spec/model/group_spec.rb spec/model/subgroup_spec.rb -- -c -fs Migel::Model::Group when initialized with migel code 15 migel code should be 15 subgroups should be empty limitation_text.to_s should be '' name.to_s should be '' name.xx should be nil Migel::Model::Group when limitation_text is updated limitation_text.de should be 'limitation text' Migel::Model::Subgroup when initialized with code 22 (group_code: 11) code should be 22 migel_code should be 11.22 migelids should be empty parent should be group limitation_text.to_s should be '' name.to_s should be '' name.xx should be nil Migel::Model::Group when limitation_text is updated limitation_text.de should be 'limitation text' Finished in 0.00627 seconds 14 examples, 0 failures
Result
Reference
- http://akr97.com/capsule/?p=1270 (Japanese)
migel server side
- model
group.rb(coverage: 100%)subgroup.rb(coverage: 100%)migelid.rb(coverage: 100%)product.rb(coverage: 100%)
- util
- swissindex.rb
- server.rb
- multilingual.rb