Must investigate why I have many occurrences of "Duplicate name" for E-Mail in /service/virbac.bbmb.ch/log/main/current while running virbac_migrate_to_utf_8. Idea: list all e-mail address when starting virbac.bbmb.ch and virbac.bbmb.ch-yus daemons.
This can be done by adding in bin/yusd the lines
puts Yus::Entity.odba_extent.collect{ |x| x.name}
logger.info('All name (aka e-mail)') { sprintf('%s', Yus::Entity.odba_extent.collect{ |x| x.name}.join("\n")) }
Dropping and reloading now both virbac_bbmb and virbac_yus. Running the following scripts:
sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee virbac_yus_1.log sudo -u bbmb /usr/local/bin/ruby231 ./yus_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee migrate_virbac_yus.log sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee virbac_yus_2.log sudo -u bbmb /usr/local/bin/ruby231 ./virbac_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/config.yml 2>&1 | tee migrate_virbac.log # as it hangs we restart it sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/virbac.bbmb.ch/etc/yus.yml 2>&1 | tee virbac_yus_3.log sudo -u bbmb /usr/local/bin/ruby231 ./virbac_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/config.yml 2>&1 | tee migrate_virbac_2.log
In all the generated log files I did not find any occurrence of Duplicate. But I am still concerned about UTF-8 errors. Therefore I called grep -i arzt-gross *log and found.
migrate_virbac.log:force_encoding from ISO_8859_1 ASCII-8BIT. is now info@tierarzt-grosshöchstetten.ch virbac_yus_1.log:info@tierarzt-grosshöchstetten.ch virbac_yus_2.log:info@tierarzt-grosshöchstetten.ch virbac_yus_3.log:info@tierarzt-grosshöchstetten.ch
It looks for me as something went wrong. Running grep '@' virbac_yus_3.log | sort made clear, that all E-Mail addresses appear twice. Eg. grep -n urswetli@gmx.ch virbac_yus_*.log returns
virbac_yus_1.log:105:urswetli@gmx.ch virbac_yus_1.log:1153:urswetli@gmx.ch virbac_yus_2.log:76:urswetli@gmx.ch virbac_yus_2.log:1125:urswetli@gmx.ch virbac_yus_3.log:76:urswetli@gmx.ch virbac_yus_3.log:1125:urswetli@gmx.ch
But this is a problem already present in the database coming from fastpower. Other funny e-mail addresses are:
pius.rechsteiner@bvet.admin.ch/markus.thony@bluewi praxis@tierärzte-team.ch tierärzteteam24@bluewin.ch info@tierarzt-grosshöchstetten.ch
When logging in as an admin user. I have the following UTF-8 Problems. Under http://virbac.bbmb.ngiger.ch/de/bbmb/customers/filter/gross the Kundenname is "Fässler" instead of "Fässler". The Ort is "Grosshöchstetten" instead of "Grosshöchstetten" and the Email is "info@tierarzt-grosshöchstetten.ch" instead of "info@tierarzt-grosshöchstetten.ch".
But in the migrate_virbac_yus.log I found force_encoding from ISO_8859_1 ASCII-8BIT. is now info@tierarzt-grosshöchstetten.ch. Dropping and reloading the virbac_yus db. Breaking into pry at this point.
Breaking (similar e-mail) I have
[24] pry(#<Yus::Server>)> child => "praxis@tier\xC3\xA4rzte-team.ch" [26] pry(#<Yus::Server>)> puts child.force_encoding(Encoding::ISO_8859_1) praxis@tierärzte-team.ch [33] pry(#<Yus::Server>)> child.encoding => #<Encoding:ISO-8859-1>
Created a file test.rb with the following content
Changing line 2 in yus_migrate_to_utf_8 from UTF-8 to # encoding: ASCII-8BIT fixed the problem.
But the columns name, ort and e-mail of http://virbac.bbmb.ngiger.ch/de/bbmb/customers/filter/gross still contain errors. Also changing the encoding of virbac_migrate_to_utf_8. Reloading the virbac_bbmb database again and running virbac_migrate_to_utf_8. This fixed the problem.
Pushed commits
Doing some manual tests. Found at that I cannot in the Schnellbestellung to add an item which has a umlaut (Hexarinse Zahnspülung 237 ml). Tried to search only with Zahn and got from http://virbac.bbmb.ngiger.ch/de/bbmb/search_favorites/query/Zahn the error Encoding::CompatibilityError incompatible character encodings: ASCII-8BIT and UTF-8. Maybe only the indices are wrong. How can I rebuild them?
Also the csv_importer fails with the following backtrace
sv_importer.rb: importing /var/www/virbac.bbmb.ch/var/ftp/KUNDEN.CSV
create_entity DRb::DRbObject
rename DRb::DRbObject
...rename DRb::DRbObject
rename DRb::DRbObject
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/mail.rb:60 Suppress sending mail with subject: Bbmb(Virbac): Unknown Entity 'info@tierarzt-grosshöchstetten.ch'
from orders.virbac@bbmb.ch to: ["ngiger@ywesee.com"] cc: [] reply_to:
Yus::UnknownEntityError
Unknown Entity 'info@tierarzt-grosshöchstetten.ch'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:182:in `find_or_fail'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:244:in `block in rename'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:243:in `synchronize'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/session.rb:243:in `rename'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1625:in `perform_without_block'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1584:in `perform'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
(druby://localhost:12002) /usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/server.rb:94:in `block in rename_user'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1625:in `perform_without_block'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1584:in `perform'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12002) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:11:in `block_yield'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:18:in `block in perform_with_block'
(druby://localhost:12003) /usr/local/lib/ruby/gems/2.3.0/gems/yus-1.0.4/lib/yus/server.rb:22:in `autosession'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/invokemethod.rb:15:in `perform_with_block'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1582:in `perform'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1658:in `block (2 levels) in main_loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `loop'
(druby://localhost:12003) /usr/local/lib/ruby/2.3.0/drb/drb.rb:1654:in `block in main_loop'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/util/server.rb:88:in `rename_user'
/usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:65:in `email='
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:87:in `block in import_record'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:78:in `each'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:78:in `import_record'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:30:in `block in import'
/usr/local/lib/ruby/2.3.0/csv.rb:1748:in `each'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:25:in `each_with_index'
/var/www/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:25:in `import'
After setting a pry debug statement, I found that import script dropped the umlauf from the name as seen by this snippet (inside method rename_user of bbmb/lib/bbmb/util/server.rb.
[1] pry(#<BBMB::Util::Server>)> new_name => "info@tierarzt-grosshchstetten.ch" [2] pry(#<BBMB::Util::Server>)> old_name => "info@tierarzt-grosshöchstetten.ch"
I think virbac.bbmb.ch needs to modify its csv_importer.rb to contain a line # encoding: ASCII-8BIT, too. Problem got fixed by changing
csv = CSV.new(File.read(io.to_path).encode('UTF-8', {invalid: :replace, undef: :replace, replace: ''}))
to
csv = CSV.new(File.read(io.to_path, :encoding => 'iso-8859-1').encode('UTF-8'))
Now I have duplicated names, e.g. I have
[15] pry(#<BBMB::Util::CustomerImporter>)> Model::Customer.odba_extent.find_all{|x| /docdahn/.match(x.email)}
=> [#<BBMB::Model::Customer:0x000000034249a8
@address1=nil,
@address2="9, Les Grands-Champs",
@archive=#<ODBA::Stub:27337320#2688 @odba_class=Hash @odba_container=27337940#2687>,
@canton="VD",
@city="Bogis-Bossey",
@current_order=#<ODBA::Stub:27337700#228973 @odba_class=BBMB::Model::Order @odba_container=27337940#2687>,
@customer_id="769656",
@email="docdahn@bluewin.ch",
@favorites=#<ODBA::Stub:27337840#2689 @odba_class=BBMB::Model::Order @odba_container=27337940#2687>,
@language="fr",
@odba_id=2687,
@odba_observers=[],
@odba_persistent=true,
@organisation="Dahn Michel",
@plz="1279",
@protected={},
@quotas=#<ODBA::Stub:27337480#2690 @odba_class=Array @odba_container=27337940#2687>,
@status=:active>]
# while import the record
[20] pry(#<BBMB::Util::CustomerImporter>)> record
=> ["3064", "A", "Dahn", "", "1297", "Founex", "Case postale 47", "docdahn@bluewin.ch", "F ", "VD", nil]
# as
[21] pry(#<BBMB::Util::CustomerImporter>)> Model::Customer.find_by_customer_id(customer_id)
=> nil
23] pry(#<BBMB::Util::CustomerImporter>)> customer_id
=> "3064"
Fixed this problem by
But now I have problems handling two customer with the same email. Found it
grep chpatfrei *CSV KUNDEN.CSV:3114,"A","Frei","","8371","Busswil TG","Hauptstrasse 21","chpatfrei@bluewin.ch","D ","TG", KUNDEN.CSV:3138,"L","Frei Urs","Tierarzt","5076","B�zen","Heuigstrasse 161","chpatfrei@bluewin.ch","D ","AG2",
Fixing the error handling of csv_importer.rb as the returned error has changed.
Pushed a commit which prevented publishing the yus gem. See Fixed case for allowed_push_host
Must fix the following problem when importing ABSCHLUSS.CSV
NoMethodError undefined method `article_number' for nil:NilClass /usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:58:in `block in quota' /usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `each' /usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `find' /usr/local/lib/ruby/gems/2.3.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `method_missing' /usr/local/lib/ruby/gems/2.3.0/gems/bbmb-2.0.5/lib/bbmb/model/customer.rb:58:in `quota' /home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:268:in `import_record' /home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:34:in `block in import' /usr/local/lib/ruby/2.3.0/csv.rb:1748:in `each' /home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:27:in `each_with_index' /home/niklaus/git/virbac.bbmb.ch/lib/bbmb/util/csv_importer.rb:27:in `import'
Also had to made lib/bbmb/util/updater.rb more robust by changing the sprintf to sprintf("i entities", importer.to_i, count.to_i). As Ruby 2.3 will throw an error if passing a nil for '%i' parameter.
Pushed commits:
The problem when searching for a product with an umlaut has vanished. http://virbac.bbmb.ngiger.ch/de/bbmb/search/query/Hexarinse shows exactly one product Hexarinse Zahnspülung 237 ml. Searching for Zahn returns 3 products, two of them contain an Umlaut.
Resuming: The following two steps must me made to migrate virbac.bbmb.ch to UTF-8 and Ruby 2.3.1
cd /checkout/of/yus && sudo -u bbmb /usr/local/bin/ruby231 yus_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/yus.yml
cd /var/www/virbac.bbmb.ch && sudo -u bbmb /usr/local/bin/ruby231 virbac_migrate_to_utf_8 config=/var/www/virbac.bbmb.ch/etc/config.yml
Added a script to migrate the yus db for vetoquinol called bbmb_yus. It uses ydim-inject for its bbmb data.
Added a line mail_suppress_sending: true to /var/www/vetoquinol.bbmb.ch/etc/config.yml
To migrate the vetoquinol yus I called sudo -u bbmb /usr/local/bin/ruby231 ./yus_migrate_to_utf_8 config=/var/www/vetoquinol.bbmb.ch/etc/yus.yml, which migrated 293 entities. None of them contained an Umlaut in its e-mail address.
Changing /service/vetoquinol.bbmb.ch-yus/run to
#!/bin/sh exec 2>&1 exec sudo -u bbmb /usr/local/bin/ruby231 /usr/local/bin/yusd config=/var/www/vetoquinol.bbmb.ch/etc/yus.yml
This workd and starts a DRB session on port 12001.
Changed in /var/www/vetoquinol.bbmb.ch/etc/ydim.yml the server_url to 'druby://127.0.0.1:12375. On this port 12375 there is a ydimd listening.
Could not find any product which contains an umlaut.
Toggling "Zusatzinformation" under http://vetoquinol.bbmb.ngiger.ch/de/bbmb (when logged in) works.
Toggling "Bestellen Sie online.." under http://vetoquinol.bbmb.ngiger.ch/de/bbmb/logout/ does not work. Why? Because it does not work under http://vetoquinol.bbmb.ch/de/bbmb/logout. Therefore I think we reached feature parity.
Added dojo. Pushed commits: