Running Migel::Model::Product.all.find_all{|x| /Bauerfeind/i.match(x.company_name.send(:de)) }.each{|x| x.delete } via bin/admin takes a long time. Therefore we just change the status to inactiv ('I') via Migel::Model::Product.all.find_all{|x| /Bauerfeind AG/.match(x.company_name.send(:fr)) }.each{|x| x.status = 'I' }. This is much faster, and should do the job, too. Updated the spec test for it.
Reloading migel db to test the script.
Must also correct the error, where Bauerfeind has no Pharmacode in its CSV but the text "nicht erfasst". Fixed the error, by search by ean_code if the pharmacode is not numeric. Must add this criteria also when searching for a matching pharmacode.
Reloading migel db to test the script.
On thinpower I get the 1640 active Bauerfeind products via
migel> Migel::Model::Product.all.find_all{|x| x.status == 'A' && /bauerfeind/i.match(x.companyname.to_s)}.size
-> 1640
On oddb-ci2 the same expression return 1672. But I have 1771 lines in the CSV file. Why do I miss 99 lines? Some migel code are ignored, but show only two GenoTrain items
nicht erfasst 4046445871405 GenuTrain S Pro, titan, links, 7 A Bauerfeind AG 279.00 G Twitter tell a friend nicht erfasst 4046445481949 GenuTrain S Pro, titan, rechts, 1 A Bauerfeind AG 279.00 G Twitter tell a friend
Why is 4046445481963 "GenuTrain S Pro,titan,rechts,3" noch added? I think I must use the ean_code when the pharmacode is not a valid digital number. Else we the several products get matched to the non existing pharmacode 'nicht erfasst'.
Now many things look better, but I still have problems, eg. that I have a product with ean_13 4026358052542 and pharmacode "nicht erfasst" 'VenoTrain soft, AG, Haftband Noppe, Ccl 2, geschlossene Fußspitze, S normal short, natur'.
Creating a test files using File.open('ausgabe.tst', 'w+') { |f| Migel::Model::Product.all.find_all{|x| x.status == 'A' && /bauerfeind/i.match(x.companyname.to_s)}.each{|p| f.puts "#{p.ean_code},#{p.pharmacode},#{p.status},#{p.ppub},#{p.article_name.de}" } }
Using the EAN-13 value as pharmacode where the pharmacode helps. As we cannot change the pharmacode of an existing product (this would lead to inconsistencies in the database) I must delete these products manually via bin/admin.
Looking at http://oddb-ci2.dyndns.org/de/gcc/search/zone/migel/search_query/GENUTRAIN I still see an entry for "4046445107818 GENUTRAIN P3 Kniebandage links titan Gr 5 40-43cm", which does not appear in the CSV file. BUt I can find it via the pharmacode 2244410 "11041403080705;4046445600203;GenuTrain P3,titan,links,5;GenuTrain P3,titane,gauche,5;2244410;05.04.05.00.1; Fr. 160.60 ;GenuTrain P3,titan,links,5;GenuTrain P3,titane,gauche,5", where it is in the migel_group '05.04.03.00.1' != csv '05.04.05.00.1'
Must rework the csv_importer to handle this case, tomorrow.
Running sudo -u apache jobs/update_migel produces the following error
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:588:in
`gsub': invalid byte sequence in UTF-8 (ArgumentError)
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:588:in
`update_fulltext_index'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:390:in
`do_update_index'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:71:in
`block (3 levels) in fill'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:70:in
`each'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:70:in
`block (2 levels) in fill'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:69:in
`each'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:69:in
`block in fill'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:66:in
`each'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/index.rb:66:in
`fill'
from /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:335:in
`fill_index'
from /var/www/migel/lib/migel/util/server.rb:328:in
`rebuild_fulltext_index_table'
from /var/www/migel/lib/migel/util/server.rb:387:in
`rebuild_fulltext_index_tables'
from /var/www/migel/lib/migel/util/server.rb:279:in
`init_fulltext_index_tables'
from jobs/update_migel:14:in `block in <module:Util>'
from /var/www/migel/lib/migel/util/job.rb:22:in `call'
from /var/www/migel/lib/migel/util/job.rb:22:in `run'
from jobs/update_migel:12:in `<module:Util>'
from jobs/update_migel:11:in `<module:Migel>'
from jobs/update_migel:10:in `<main>'
Trying to force this error with the following bin/admin commands
$res = all_migelids.values.collect{ |x| x.full_description(:de) }.join("\n")
$res = all_migelids.values.collect{ |x| x.full_description(:fr) }.join("\n")
$res = all_products.values.collect{ |x| x.full_description(:de) }.join("\n")
$res = all_products.values.collect{ |x| x.full_description(:fr) }.join("\n")
Statement executed successfully on thinpower. Errors stems from the call to rebuild_fulltext_index_table(index_definition_migel_fr) which resolves to all_migelids.values.full_description(:fr). Also all encoding seem to be UTF-8, as seen by
$res = all_products.values.collect{ |x| x.full_description(:fr).encoding }.uniq
-> [#<Encoding:UTF-8>]
Calling manually via bin/admin rebuild_fulltext_index_tables to see whether I can reproduce the error show above. Did not throw any error. Don't know how what caused the error.