view · edit · sidebar · attach · print · history

Index>

20160712-virbac-csv-import

Summary

  • Fix CSV import for virbac
  • Migrate vetoquinol to Ruby 2.3.1
  • Keep in Mind

Commits

Index

Fix CSV import for virbac

Must investigate why I have many occurrences of "Duplicate name" for E-Mail in /service/virbac.bbmb.ch/log/main/current while running virbac_migrate_to_utf_8. Idea: list all e-mail address when starting virbac.bbmb.ch and virbac.bbmb.ch-yus daemons.

This can be done by adding in bin/yusd the lines

  puts Yus::Entity.odba_extent.collect{ |x| x.name}
  logger.info('All name (aka e-mail)') { sprintf('%s', Yus::Entity.odba_extent.collect{ |x| x.name}.join("\n")) }

Dropping and reloading now both virbac_bbmb and virbac_yus. Running the following scripts:

  sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee virbac_yus_1.log
  sudo -u bbmb /usr/local/bin/ruby231 ./yus_migrate_to_utf_8  config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee migrate_virbac_yus.log
  sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee virbac_yus_2.log
  sudo -u bbmb /usr/local/bin/ruby231 ./virbac_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/config.yml 2>&1 | tee migrate_virbac.log
  # as it hangs we restart it
  sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee virbac_yus_3.log
  sudo -u bbmb /usr/local/bin/ruby231 ./virbac_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/config.yml 2>&1 | tee migrate_virbac_2.log

In all the generated log files I did not find any occurrence of Duplicate. But I am still concerned about UTF-8 errors. Therefore I called grep -i arzt-gross *log and found.

  migrate_virbac.log:force_encoding from ISO_8859_1 ASCII-8BIT. is now info@tierarzt-grosshöchstetten.ch
  virbac_yus_1.log:info@tierarzt-grosshöchstetten.ch
  virbac_yus_2.log:info@tierarzt-grosshöchstetten.ch
  virbac_yus_3.log:info@tierarzt-grosshöchstetten.ch

It looks for me as something went wrong. Running grep '@' virbac_yus_3.log | sort made clear, that all E-Mail addresses appear twice. Eg. grep -n urswetli@gmx.ch virbac_yus_*.log returns

  virbac_yus_1.log:105:urswetli@gmx.ch
  virbac_yus_1.log:1153:urswetli@gmx.ch
  virbac_yus_2.log:76:urswetli@gmx.ch
  virbac_yus_2.log:1125:urswetli@gmx.ch
  virbac_yus_3.log:76:urswetli@gmx.ch
  virbac_yus_3.log:1125:urswetli@gmx.ch

But this is a problem already present in the database coming from fastpower. Other funny e-mail addresses are:

  pius.rechsteiner@bvet.admin.ch/markus.thony@bluewi
  praxis@tierärzte-team.ch
  tierärzteteam24@bluewin.ch
  info@tierarzt-grosshöchstetten.ch

When logging in as an admin user. I have the following UTF-8 Problems. Under http://virbac.bbmb.ngiger.ch/de/bbmb/customers/filter/gross the Kundenname is "Fässler" instead of "Fässler". The Ort is "Grosshöchstetten" instead of "Grosshöchstetten" and the Email is "info@tierarzt-grosshöchstetten.ch" instead of "info@tierarzt-grosshöchstetten.ch". But in the migrate_virbac_yus.log I found force_encoding from ISO_8859_1 ASCII-8BIT. is now info@tierarzt-grosshöchstetten.ch. Dropping and reloading the virbac_yus db. Breaking into pry at this point.

Breaking (similar e-mail) I have

[24] pry(#<Yus::Server>)> child
=> "praxis@tier\xC3\xA4rzte-team.ch"
[26] pry(#<Yus::Server>)> puts child.force_encoding(Encoding::ISO_8859_1)
praxis@tierärzte-team.ch
[33] pry(#<Yus::Server>)> child.encoding
=> #<Encoding:ISO-8859-1>

Created a file test.rb with the following content

Changing line 2 in yus_migrate_to_utf_8 from UTF-8 to # encoding: ASCII-8BIT fixed the problem.

But the columns name, ort and e-mail of http://virbac.bbmb.ngiger.ch/de/bbmb/customers/filter/gross still contain errors. Also changing the encoding of virbac_migrate_to_utf_8. Reloading the virbac_bbmb database again and running virbac_migrate_to_utf_8. This fixed the problem.

Pushed commits

Doing some manual tests. Found at that I cannot in the Schnellbestellung to add an item which has a umlaut (Hexarinse Zahnspülung 237 ml). Tried to search only with Zahn and got from http://virbac.bbmb.ngiger.ch/de/bbmb/search_favorites/query/Zahn the error Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8. Maybe only the indices are wrong. How can I rebuild them?

Also the csv_importer fails with the following backtrace

sv_importer.rb: importing /var/www/virbac.bbmb.ch/var/ftp/KUNDEN.CSV
create_entity DRb::DRbObject
rename DRb::DRbObject
...rename DRb::DRbObject
rename DRb::DRbObject
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/mail.rb:60 Suppress sending mail with subject: Bbmb(Virbac): Unknown Entity 'info@tierarzt-grosshöchstetten.ch'
    from orders.virbac@bbmb.ch to: ["ngiger@ywesee.com"] cc: [] reply_to: 
Yus::UnknownEntityError
Unknown Entity 'info@tierarzt-grosshöchstetten.ch'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:182:in `find_or_fail'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:244:in `block in rename'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:243:in `synchronize'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:243:in `rename'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1625:in `perform_without_block'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1584:in `perform'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
(druby://localhost:12002) /usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/server.rb:94:in `block in rename_user'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1625:in `perform_without_block'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1584:in `perform'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:11:in `block_yield'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:18:in `block in perform_with_block'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/server.rb:22:in `autosession'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:15:in `perform_with_block'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1582:in `perform'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/server.rb:88:in `rename_user'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:65:in `email='
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:87:in `block in import_record'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:78:in `each'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:78:in `import_record'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:30:in `block in import'
/usr/local/lib/ruby/2.3.0/csv.rb:1748:in `each'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:25:in `each_with_index'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:25:in `import'

After setting a pry debug statement, I found that import script dropped the umlauf from the name as seen by this snippet (inside method rename_user of bbmb/lib/bbmb/util/server.rb.

[1] pry(#<BBMB::Util::Server>)> new_name
=> "info@tierarzt-grosshchstetten.ch"
[2] pry(#<BBMB::Util::Server>)> old_name
=> "info@tierarzt-grosshöchstetten.ch"

I think virbac.bbmb.ch needs to modify its csv_importer.rb to contain a line # encoding: ASCII-8BIT, too. Problem got fixed by changing

  csv = CSV.new(File.read(io.to_path).encode('UTF-8', {invalid: :replace, undef: :replace, replace: ''}))

to

  csv = CSV.new(File.read(io.to_path, :encoding => 'iso-8859-1').encode('UTF-8'))

Now I have duplicated names, e.g. I have

[15] pry(#<BBMB::Util::CustomerImporter>)> Model::Customer.odba_extent.find_all{|x| /docdahn/.match(x.email)}
=> [#<BBMB::Model::Customer:0x000000034249a8
  @address1=nil,
  @address2="9, Les Grands-Champs",
  @archive=#<ODBA::Stub:27337320#2688 @odba_class=Hash @odba_container=27337940#2687>,
  @canton="VD",
  @city="Bogis-Bossey",
  @current_order=#<ODBA::Stub:27337700#228973 @odba_class=BBMB::Model::Order @odba_container=27337940#2687>,
  @customer_id="769656",
  @email="docdahn@bluewin.ch",
  @favorites=#<ODBA::Stub:27337840#2689 @odba_class=BBMB::Model::Order @odba_container=27337940#2687>,
  @language="fr",
  @odba_id=2687,
  @odba_observers=[],
  @odba_persistent=true,
  @organisation="Dahn Michel",
  @plz="1279",
  @protected={},
  @quotas=#<ODBA::Stub:27337480#2690 @odba_class=Array @odba_container=27337940#2687>,
  @status=:active>]
# while import the record
[20] pry(#<BBMB::Util::CustomerImporter>)> record
=> ["3064", "A", "Dahn", "", "1297", "Founex", "Case postale 47", "docdahn@bluewin.ch", "F ", "VD", nil]
# as
[21] pry(#<BBMB::Util::CustomerImporter>)> Model::Customer.find_by_customer_id(customer_id)
=> nil
23] pry(#<BBMB::Util::CustomerImporter>)> customer_id
=> "3064"

Fixed this problem by

  • Searching for a customer by email if it cannot be found by customer_id
  • not setting the email if it equals the the existing

But now I have problems handling two customer with the same email. Found it

grep chpatfrei *CSV
KUNDEN.CSV:3114,"A","Frei","","8371","Busswil TG","Hauptstrasse 21","chpatfrei@bluewin.ch","D ","TG",
KUNDEN.CSV:3138,"L","Frei Urs","Tierarzt","5076","B&#65533;zen","Heuigstrasse 161","chpatfrei@bluewin.ch","D ","AG2",

Fixing the error handling of csv_importer.rb as the returned error has changed.

Pushed a commit which prevented publishing the yus gem. See Fixed case for allowed_push_host

Must fix the following problem when importing ABSCHLUSS.CSV

NoMethodError
undefined method `article_number' for nil:NilClass
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:58:in `block in quota'
/usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `each'
/usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `find'
/usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `method_missing'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:58:in `quota'
/home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:268:in `import_record'
/home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:34:in `block in import'
/usr/local/lib/ruby/2.3.0/csv.rb:1748:in `each'
/home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:27:in `each_with_index'
/home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:27:in `import'

Also had to made lib/bbmb/util/updater.rb more robust by changing the sprintf to sprintf("i entities", importer.to_i, count.to_i). As Ruby 2.3 will throw an error if passing a nil for '%i' parameter.

Pushed commits:

The problem when searching for a product with an umlaut has vanished. http://virbac.bbmb.ngiger.ch/de/bbmb/search/query/Hexarinse shows exactly one product Hexarinse Zahnspülung 237 ml. Searching for Zahn returns 3 products, two of them contain an Umlaut.

Resuming: The following two steps must me made to migrate virbac.bbmb.ch to UTF-8 and Ruby 2.3.1

  • cd /checkout/of/yus && sudo -u bbmb /usr/local/bin/ruby231 yus_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/yus.yml
  • cd /var/www/virbac.bbmb.ch && sudo -u bbmb /usr/local/bin/ruby231 virbac_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/config.yml

Migrate vetoquinol to Ruby 2.3.1

Added a script to migrate the yus db for vetoquinol called bbmb_yus. It uses ydim-inject for its bbmb data.

Added a line mail_suppress_sending: true to /var/www/vetoquinol.bbmb.ch/etc/config.yml

To migrate the vetoquinol yus I called sudo -u bbmb /usr/local/bin/ruby231 ./yus_migrate_to_utf_8 config=/var/www/vetoquinol.bbmb.ch/etc/yus.yml, which migrated 293 entities. None of them contained an Umlaut in its e-mail address.

Changing /service/vetoquinol.bbmb.ch-yus/run to

#!/bin/sh
exec 2>&1
exec sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/vetoquinol.bbmb.ch/etc/yus.yml

This workd and starts a DRB session on port 12001.

Changed in /var/www/vetoquinol.bbmb.ch/etc/ydim.yml the server_url to 'druby://127.0.0.1:12375. On this port 12375 there is a ydimd listening.

Could not find any product which contains an umlaut.

Toggling "Zusatzinformation" under http://vetoquinol.bbmb.ngiger.ch/de/bbmb (when logged in) works.

Toggling "Bestellen Sie online.." under http://vetoquinol.bbmb.ngiger.ch/de/bbmb/logout/ does not work. Why? Because it does not work under http://vetoquinol.bbmb.ch/de/bbmb/logout. Therefore I think we reached feature parity.

Added dojo. Pushed commits:

view · edit · sidebar · attach · print · history
Page last modified on July 12, 2016, at 04:55 PM