view · edit · sidebar · attach · print · history

< Niklaus.20160817-restart-migel-after-import_bauerfeind | Index | Niklaus.20160815-delete-bauernfeind-before-import >>

20160816-delete-bauernfeind-before-import

Summary

  • Before importing Bauerfeind products from CSV all Bauerfeind products must be deleted
  • Add update_evidentia_fi_search_links to import_daily
  • Keep in Mind

Commits

Index

Before importing Bauerfeind products from CSV all Bauerfeind products must be deleted

Two problems are remaining:

  • rework the csv_importer to handle the case that a product has a new migel code
  • jobs/update_migel hangs when deleting

Started yesterday at 21:30 jobs/update_migel. Last line see is 2016-08-15 21:40:31 +0200: update_group ["99", "50", "01", "00", "1"] groupcd 99 de. Looking for jobs

niklaus    523  2853  0 08:19 pts/2    00:00:00 grep --color=auto migel
root      1875  1835  0 Aug15 ?        00:00:00 supervise migeld
niklaus  17023  2879  0 Aug15 pts/3    00:00:00 migel (Admin-Tool)
root     17303  1875  0 Aug15 ?        00:00:00 sudo -u apache /usr/local/bin/ruby bin/migeld
apache   17305 17303  0 Aug15 ?        00:01:43 /usr/local/bin/ruby bin/migeld
postgres 17308  1774  0 Aug15 ?        00:00:02 postgres: migel migel [local] idle
postgres 17309  1774  0 Aug15 ?        00:00:02 postgres: migel migel [local] idle
postgres 17310  1774  0 Aug15 ?        00:00:02 postgres: migel migel [local] idle
postgres 17311  1774  0 Aug15 ?        00:00:02 postgres: migel migel [local] idle
postgres 17312  1774  0 Aug15 ?        00:00:02 postgres: migel migel [local] idle
root     22277  2409  0 Aug15 pts/1    00:00:00 sudo -u apache jobs/update_migel
niklaus  22278  2409  0 Aug15 pts/1    00:00:00 tee /home/niklaus/git/migel/update_migel.log
apache   22281 22277  4 Aug15 pts/1    00:28:15 /usr/local/bin/ruby jobs/update_migel
postgres 22284  1774  0 Aug15 ?        00:04:59 postgres: migel migel [local] idle
postgres 22285  1774  0 Aug15 ?        00:05:00 postgres: migel migel [local] idle
postgres 22286  1774  0 Aug15 ?        00:05:00 postgres: migel migel [local] DELETE
postgres 22287  1774  0 Aug15 ?        00:04:59 postgres: migel migel [local] idle
postgres 22288  1774  0 Aug15 ?        00:05:00 postgres: migel migel [local] idle

There is a postgres DELETE statement which hangs. I think a call to delete for a product does not work anymore, as yesterday Migel::Model::Product.all.find_all{|x| /Bauerfeind/i.match(x.company_name.send(:de)) }.each{|x| x.delete } did not finish, too. Trying to delete a singel product via bin/admin (after killing the import and restarting the migeld).

The following lines must be checked 11041403089601;4046445600227;GenuTrain P3 mit SR,titan,re,1;GenuTrain P3,silicone,titane,droit,1;6509897;05.04.05.00.1; Fr. 179.20 ;GenuTrain P3 mit SR,titan,re,1;GenuTrain P3,silicone,titane,droit,1 11041403080601;4046445600104;GenuTrain P3,titan,rechts,1;GenuTrain P3,titane,droit,1;2244775;05.04.05.00.1; Fr. 160.60 ;GenuTrain P3,titan,rechts,1;GenuTrain P3,titane,droit,1

bin/admin show that the product with EAN 4046445107818 has the wrong migel_code 05.04.03.00.1

migel> Migel::Model::Product.all.find{|x| '4046445107818'.eql?(x.ean_code)}.migel_code
-> 05.04.03.00.1
migel> Migel::Model::Product.all.find{|x| '4046445107818'.eql?(x.ean_code)}
-> GENUTRAIN P3 Kniebandage links titan Gr 5 40-43cm
migel> Migel::Model::Product.all.find{|x| '4046445107818'.eql?(x.ean_code)}.status
-> A
migel> Migel::Model::Product.all.find{|x| '4046445107818'.eql?(x.ean_code)}.company_name
-> Bauerfeind AG
migel> Migel::Model::Product.all.find{|x| '4046445107818'.eql?(x.ean_code)}.company_name.send(:de)
-> Bauerfeind AG
migel> Migel::Model::Product.all.find{|x| '4046445107818'.eql?(x.ean_code)}.company_name.send(:fr)
-> Bauerfeind AG

Looks, as I have forgotton to save the product, after changing its status. Calling Migel::Model::Product.all.find{|x| x.ean_code.eql?('4046445419065')}.delete hangs, too. Replacing the delete by status = 'I' to deactivate it.

Added debug output for deletes. They did not show up. Running jobs/import_migel_bauerfeind always takes over 13 minutes to get the interesting code parts.

After running http://oddb-ci2.dyndns.org/de/gcc/search/zone/migel/search_query/4046445419065 shows two products, one with status 'I' and '05.04.02.00.1', the other one with status 'A' and ) MiGel Code: '05.04.05.00.1'. This is okay. Only the one with status 'A' shows up after searching for genutrain via http://oddb-ci2.dyndns.org/de/gcc/search/zone/migel/search_query/GenuTrain?

Preparing commit for push. Eliminating superfluos puts statements. Running it again. Remarked that the field "Packungsgrösse" is empty after an import for 4046445419065, as it is not present inte CSV file from Bauerfeind.

After running the import, the old entry with pharmacode '4313351' is correctly marked as inactive, has the correct migel_code and article_name. But it does not show up unter http://oddb-ci2.dyndns.org/de/gcc/migel_search/migel_code/17.03.02.00.1. Why?

Found out I can delete a product from the migel_id

$id = Migel::Model::Migelid.find_by_migel_code('17.03.02.00.1')
-> Paar
migel> $id.products.find_all{|x| x.ean_code.eql?('4046445241024')}.size
-> 2
migel> $id.products.values.delete_if{|x| x.ean_code.eql?('4046445241024') && x.status.eql?('I')}
-> Array
migel> $id.products.find_all{|x| x.ean_code.eql?('4046445241024')}.size
-> 1

Remarked that after running import_bauerfeind, I have problems calling Migel::Model::Migelid.all in the migel/bin/admin

migel> Migel::Model::Migelid.all
-> incompatible character encodings: UTF-8 and ASCII-8BIT

After reloading the database from thinpower I have

migel> Migel::Model::Migelid.all.first.name.to_s.encoding
-> ASCII-8BIT
migel> Migel::Model::Migelid.all.first.name.to_s
-> Gripper für Port-A-Cath

Calling Migel::Model::Migelid.all on thinpower takes several minutes.

Running migrate_utf8 on oddb-ci2 gives

  ODBA::Stub was unable to replace Hash#32292 from Migel::Model::Migelid:#58
  ODBA::Stub was unable to replace Hash#32293 from Migel::Model::Migelid:#880

On thinpower the Migel::Model::Migelid.all.first.name.to_s.encoding returns UTF-8.

Checked that the migel-db on oddb-ci2 is really UTF-8 with

 psql -U apache migel --host=localhost -c 'SHOW SERVER_ENCODING'
 server_encoding 
-----------------
 UTF8
(1 Zeile)

Deleting all products from Bauerfeind works with this method

     def clear_bauerfeind_products
        @nr_products_before = get_nr_active_bauerfeind_products
        items = 0
        Migel::Model::Migelid.all.each do |migel_id|
            migel_code = migel_id.migel_code
            nr_invalids = migel_id.products.find_all{|x| /Bauerfeind/i.match(x.companyname) }.size
            if nr_invalids > 0
              items += nr_invalids
              puts "#{Time.now}: Total #{items}: Deleting  #{nr_invalids} items for #{migel_code} by #{company_re_de}"
              migel_id.products.delete_if{|x| /Bauerfeind/i.match(x.companyname)  }
              migel_id.save
            end
        end
        @nr_products_after = get_nr_active_bauerfeind_products
        puts "#{Time.now}: Updated via DE #{items} items"
      end

Reorganizing the import to first mark all Bauerfeind products as inactive and deleting at the end all inactive Bauerfeind products. This allows me to conserver the Packungsgrösse for existing products. Dropping and reloading the migel database before running the import.

Pushed commit Inactive Bauerfeind products before import via CSV

Add update_evidentia_fi_search_links to import_daily

Pushed commit tested over night Add update_evidentia_fi_search_links to import_daily

view · edit · sidebar · attach · print · history
Page last modified on August 17, 2016, at 07:58 AM