view · edit · sidebar · attach · print · history

20130917-export_failures-2

<< | Index | >>


Summary

  • Fix invalid byte sequence in US-ASCII in exports

Commits

Index


on iPhone the pricelink does not work

Using the iPhone the pricelinks, e.g. for Esomep don't work. Probably because we are using a different CSS for the iPhone. We are using http://dojotoolkit.org version 1.3.0. In src/util/csstemplate.rb I find some switching around for different templates like desitin or mobile. (Probably what is used with iphone?)

Executed diff /var/www/oddb.org/doc/resources/gcc/ /var/www/oddb.org/doc/resources/mobile/ on thinpower and only found diffs in font sizes.

oddbdat-export: fix Undefined method `-' for nil:NilClass

  • In export of September 15 we got the following error mail "Error in export oddbdat"
  • Modified src/util/exporter.rb to run only export_oddbdat. Started on thinpower.
  • Added some debug info to ext/export/src/oddbdat.rb and started export on my VM.

Had to kill the export after 5 hours as my and generating a 6.6 GB big 09.log file. Saved the last 100 lines.

invalid byte sequence in US-ASCII

Introduced debugging all calls to CsvExport.dump via logfile to oddb/debug. Starting it on my VM. Did not get any more output. Therefore adding Process.pid to each log and restarting all oddb-services on my VM. Moved export_index_therapeuticus_csv, export_doc_csv, export_price_history_csv to the beginning of exports to see the error much earliear (e.g. after 5 minutes instead of 23). Now it finally works on my VM. Copy csv_exporter.rb and exporter.rb to thinpower. Restarted ch.oddb-exporter service and run export_daily.

Now the error finally arrives after just one or two minutes of waiting!

2013-09-17 10:08:12 CEST9408 safe_export doctors.csv starting
2013-09-17 10:09:02 CEST9366 CsvExporter.dump 1: [:ean13, :exam, :salutation, :title, :firstname, :name, :praxis, :first_address_data, :email, :language, :specialities] item #<ODDB::Doctor:0x0000000405e200 @odba_observers=[], @origin_db=:ch, @name="Pitteloud", @firstname="Jean-Cyrille", @salutation="Herrn", @ean13="7601000013989", @praxis=false, @odba_target_ids=nil, @email="jcpitteloud@netplus.ch", @odba_prefetch=false, @exam="1987", @odba_persistent=true, @language="franz\u00F6sisch", @origin_id=28769, @oid=17959, @title="Dr. med.", @addresses=#<ODBA::Stub:33762200#27779137 @odba_class=Array @odba_container=33747200#462305>, @odba_id=462305, @pointer=:!doctor,17959., @specialities=#<ODBA::Stub:33761420#27779138 @odba_class=Array @odba_container=33747200#462305>> opts {}
2013-09-17 10:09:02 CEST9366 CsvExporter.dump 1: [:ean13, :exam, :salutation, :title, :firstname, :name, :praxis, :first_address_data, :email, :language, :specialities] item #<ODDB::Doctor:0x0000000409a6b0 @odba_observers=[], @origin_db=:ch, @name="Pattay", @firstname="Jean", @salutation="Herrn", @ean13="7601000020765", @praxis=false, @odba_target_ids=nil, @odba_prefetch=false, @exam="1949", @odba_persistent=true, @language="franz\u00F6sisch", @origin_id=9985, @oid=1522, @title="Dr. med.", @addresses=#<ODBA::Stub:33886300#27779143 @odba_class=Array @odba_container=33870680#398105>, @odba_id=398105, @pointer=:!doctor,1522., @specialities=#<ODBA::Stub:33885860#398107 @odba_class=Array @odba_container=33870680#398105>> opts {}
2013-09-17 10:09:02 CEST9366 CsvExporter.dump failed
2013-09-17 10:09:02 CEST9366 CsvExporter.dump 2: [:ean13, :exam, :salutation, :title, :firstname, :name, :praxis, :first_address_data, :email, :language, :specialities] item #<ODDB::Doctor:0x0000000409a6b0 @odba_observers=[], @origin_db=:ch, @name="Pattay", @firstname="Jean", @salutation="Herrn", @ean13="7601000020765", @praxis=false, @odba_target_ids=nil, @odba_prefetch=false, @exam="1949", @odba_persistent=true, @language="franz\u00F6sisch", @origin_id=9985, @oid=1522, @title="Dr. med.", @addresses=#<ODBA::Stub:33886300#27779143 @odba_class=Array @odba_container=33870680#398105>, @odba_id=398105, @pointer=:!doctor,1522., @specialities=#<ODBA::Stub:33885860#398107 @odba_class=Array @odba_container=33870680#398105>> opts {}

Started ruby -I/usr/lib64/ruby1.9/site_ruby -rauto_gem bin/admin. Looking for more information about failing doctor Pattay, which has OID 1522. Using

ch.oddb> doctors[1522].name
-> Pattay
 doctors[17959].name
-> Pitteloud
ch.oddb> doctors[1552].name
-> Gervasoni
ch.oddb> doctors[1522].name
-> Pattay
ch.oddb> doctors[1522]
-> #<ODDB::Doctor:0x000000158c56d0>
ch.oddb> doctors[1522].name
-> Pattay
ch.oddb> doctors[1522].addresses
-> []
ch.oddb> doctors[1522].email
->
ch.oddb> doctors[1522].specialities
-> ["Innere Medizin"]
ch.oddb> doctors[1522].name_first
-> Jean
ch.oddb> doctors[1522].salutation
-> Herrn
ch.oddb> doctors[17959].salutation
-> Herrn  
ch.oddb> doctors[17959].name  
-> Pitteloud  
ch.oddb> doctors[17959].name_first  
-> Jean-Cyrille  
ch.oddb> doctors[17959].specialities  
-> ["An\u00E4sthesiologie"]  
ch.oddb> doctors[17959].language  
-> französisch
ch.oddb> doctors[1522].language
-> französisch

I don't see anything problematic here.

Running export (only doctors) again, but forcing Rubys internal encoding to UTF-8 via sudo -u apache /usr/local/bin/ruby193 -U /var/www/oddb.org/jobs/export_daily

Difference to the file from Sept 11 is that the encoding was UTF-8 and now it is. Trying to add in src/plugin/csv_export.rb in proc export_doctors @options = { :iconv => 'UTF-8' }. That works!! Now trying whether only changing ext/export/csv_exporter.rb (in the baseclass) to add utf-8 encoding there helps.

Okay. This seems to work. Therefore diff reduced to a one-liner. Restarted export_daily to see, whether all export run now. If yes I will push my commit. Export completed successfully. Pushed Force UTF-8 encoding for CSV exports

started a new Jenkins-CI to better catch errror

I started on my private Jenkins-CI a job for oddb.org, as on travis.ci I don't get the same results as on my private machine and I want to investigate, where the problems come from.

To avoid error when running isolateds test-scripts we must use the same ruby version as the main test-suite. Fixed with commit Run isolated tests with the same ruby version

view · edit · sidebar · attach · print · history
Page last modified on September 18, 2013, at 07:54 AM