view · edit · sidebar · attach · print · history

< Niklaus.20160713-sandoz-on-fastpower | Index | Niklaus.20160711-virbac-csv-import >>



  • Fix CSV import for virbac
  • Migrate vetoquinol to Ruby 2.3.1
  • Keep in Mind



Fix CSV import for virbac

Must investigate why I have many occurrences of "Duplicate name" for E-Mail in /service/ while running virbac_migrate_to_utf_8. Idea: list all e-mail address when starting and daemons.

This can be done by adding in bin/yusd the lines

  puts Yus::Entity.odba_extent.collect{ |x|}'All name (aka e-mail)') { sprintf('%s', Yus::Entity.odba_extent.collect{ |x|}.join("\n")) }

Dropping and reloading now both virbac_bbmb and virbac_yus. Running the following scripts:

  sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/ 2>&1 | tee virbac_yus_1.log
  sudo -u bbmb /usr/local/bin/ruby231 ./yus_migrate_to_utf_8  config=/var/www/ 2>&1 | tee migrate_virbac_yus.log
  sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/ 2>&1 | tee virbac_yus_2.log
  sudo -u bbmb /usr/local/bin/ruby231 ./virbac_migrate_to_utf_8 config=/var/www/ 2>&1 | tee migrate_virbac.log
  # as it hangs we restart it
  sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/ 2>&1 | tee virbac_yus_3.log
  sudo -u bbmb /usr/local/bin/ruby231 ./virbac_migrate_to_utf_8 config=/var/www/ 2>&1 | tee migrate_virbac_2.log

In all the generated log files I did not find any occurrence of Duplicate. But I am still concerned about UTF-8 errors. Therefore I called grep -i arzt-gross *log and found.

  migrate_virbac.log:force_encoding from ISO_8859_1 ASCII-8BIT. is now info@tierarzt-grosshö

It looks for me as something went wrong. Running grep '@' virbac_yus_3.log | sort made clear, that all E-Mail addresses appear twice. Eg. grep -n virbac_yus_*.log returns

But this is a problem already present in the database coming from fastpower. Other funny e-mail addresses are:

When logging in as an admin user. I have the following UTF-8 Problems. Under the Kundenname is "Fässler" instead of "Fässler". The Ort is "Grosshöchstetten" instead of "Grosshöchstetten" and the Email is "info@tierarzt-grosshö" instead of "info@tierarzt-grosshö". But in the migrate_virbac_yus.log I found force_encoding from ISO_8859_1 ASCII-8BIT. is now info@tierarzt-grosshö Dropping and reloading the virbac_yus db. Breaking into pry at this point.

Breaking (similar e-mail) I have

[24] pry(#<Yus::Server>)> child
=> "praxis@tier\xC3\"
[26] pry(#<Yus::Server>)> puts child.force_encoding(Encoding::ISO_8859_1)
[33] pry(#<Yus::Server>)> child.encoding
=> #<Encoding:ISO-8859-1>

Created a file test.rb with the following content

Changing line 2 in yus_migrate_to_utf_8 from UTF-8 to # encoding: ASCII-8BIT fixed the problem.

But the columns name, ort and e-mail of still contain errors. Also changing the encoding of virbac_migrate_to_utf_8. Reloading the virbac_bbmb database again and running virbac_migrate_to_utf_8. This fixed the problem.

Pushed commits

Doing some manual tests. Found at that I cannot in the Schnellbestellung to add an item which has a umlaut (Hexarinse Zahnspülung 237 ml). Tried to search only with Zahn and got from the error Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8. Maybe only the indices are wrong. How can I rebuild them?

Also the csv_importer fails with the following backtrace

sv_importer.rb: importing /var/www/
create_entity DRb::DRbObject
rename DRb::DRbObject
...rename DRb::DRbObject
rename DRb::DRbObject
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/mail.rb:60 Suppress sending mail with subject: Bbmb(Virbac): Unknown Entity 'info@tierarzt-grosshö'
    from to: [""] cc: [] reply_to: 
Unknown Entity 'info@tierarzt-grosshö'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:182:in `find_or_fail'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:244:in `block in rename'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:243:in `synchronize'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:243:in `rename'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1625:in `perform_without_block'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1584:in `perform'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
(druby://localhost:12002) /usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/server.rb:94:in `block in rename_user'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1625:in `perform_without_block'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1584:in `perform'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:11:in `block_yield'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:18:in `block in perform_with_block'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/server.rb:22:in `autosession'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:15:in `perform_with_block'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1582:in `perform'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/server.rb:88:in `rename_user'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:65:in `email='
/var/www/ `block in import_record'
/var/www/ `each'
/var/www/ `import_record'
/var/www/ `block in import'
/usr/local/lib/ruby/2.3.0/csv.rb:1748:in `each'
/var/www/ `each_with_index'
/var/www/ `import'

After setting a pry debug statement, I found that import script dropped the umlauf from the name as seen by this snippet (inside method rename_user of bbmb/lib/bbmb/util/server.rb.

[1] pry(#<BBMB::Util::Server>)> new_name
=> ""
[2] pry(#<BBMB::Util::Server>)> old_name
=> "info@tierarzt-grosshö"

I think needs to modify its csv_importer.rb to contain a line # encoding: ASCII-8BIT, too. Problem got fixed by changing

  csv ='UTF-8', {invalid: :replace, undef: :replace, replace: ''}))


  csv =, :encoding => 'iso-8859-1').encode('UTF-8'))

Now I have duplicated names, e.g. I have

[15] pry(#<BBMB::Util::CustomerImporter>)> Model::Customer.odba_extent.find_all{|x| /docdahn/.match(}
=> [#<BBMB::Model::Customer:0x000000034249a8
  @address2="9, Les Grands-Champs",
  @archive=#<ODBA::Stub:27337320#2688 @odba_class=Hash @odba_container=27337940#2687>,
  @current_order=#<ODBA::Stub:27337700#228973 @odba_class=BBMB::Model::Order @odba_container=27337940#2687>,
  @favorites=#<ODBA::Stub:27337840#2689 @odba_class=BBMB::Model::Order @odba_container=27337940#2687>,
  @organisation="Dahn Michel",
  @quotas=#<ODBA::Stub:27337480#2690 @odba_class=Array @odba_container=27337940#2687>,
# while import the record
[20] pry(#<BBMB::Util::CustomerImporter>)> record
=> ["3064", "A", "Dahn", "", "1297", "Founex", "Case postale 47", "", "F ", "VD", nil]
# as
[21] pry(#<BBMB::Util::CustomerImporter>)> Model::Customer.find_by_customer_id(customer_id)
=> nil
23] pry(#<BBMB::Util::CustomerImporter>)> customer_id
=> "3064"

Fixed this problem by

  • Searching for a customer by email if it cannot be found by customer_id
  • not setting the email if it equals the the existing

But now I have problems handling two customer with the same email. Found it

grep chpatfrei *CSV
KUNDEN.CSV:3114,"A","Frei","","8371","Busswil TG","Hauptstrasse 21","","D ","TG",
KUNDEN.CSV:3138,"L","Frei Urs","Tierarzt","5076","B&#65533;zen","Heuigstrasse 161","","D ","AG2",

Fixing the error handling of csv_importer.rb as the returned error has changed.

Pushed a commit which prevented publishing the yus gem. See Fixed case for allowed_push_host

Must fix the following problem when importing ABSCHLUSS.CSV

undefined method `article_number' for nil:NilClass
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:58:in `block in quota'
/usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `each'
/usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `find'
/usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `method_missing'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:58:in `quota'
/home/niklaus/git/ `import_record'
/home/niklaus/git/ `block in import'
/usr/local/lib/ruby/2.3.0/csv.rb:1748:in `each'
/home/niklaus/git/ `each_with_index'
/home/niklaus/git/ `import'

Also had to made lib/bbmb/util/updater.rb more robust by changing the sprintf to sprintf("i entities", importer.to_i, count.to_i). As Ruby 2.3 will throw an error if passing a nil for '%i' parameter.

Pushed commits:

The problem when searching for a product with an umlaut has vanished. shows exactly one product Hexarinse Zahnspülung 237 ml. Searching for Zahn returns 3 products, two of them contain an Umlaut.

Resuming: The following two steps must me made to migrate to UTF-8 and Ruby 2.3.1

  • cd /checkout/of/yus && sudo -u bbmb /usr/local/bin/ruby231 yus_migrate_to_utf_8 config=/var/www/
  • cd /var/www/ && sudo -u bbmb /usr/local/bin/ruby231 virbac_migrate_to_utf_8 config=/var/www/

Migrate vetoquinol to Ruby 2.3.1

Added a script to migrate the yus db for vetoquinol called bbmb_yus. It uses ydim-inject for its bbmb data.

Added a line mail_suppress_sending: true to /var/www/

To migrate the vetoquinol yus I called sudo -u bbmb /usr/local/bin/ruby231 ./yus_migrate_to_utf_8 config=/var/www/, which migrated 293 entities. None of them contained an Umlaut in its e-mail address.

Changing /service/ to

exec 2>&1
exec sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/

This workd and starts a DRB session on port 12001.

Changed in /var/www/ the server_url to 'druby:// On this port 12375 there is a ydimd listening.

Could not find any product which contains an umlaut.

Toggling "Zusatzinformation" under (when logged in) works.

Toggling "Bestellen Sie online.." under does not work. Why? Because it does not work under Therefore I think we reached feature parity.

Added dojo. Pushed commits:

view · edit · sidebar · attach · print · history
Page last modified on July 12, 2016, at 04:55 PM