view · edit · sidebar · attach · print · history

Index>

0141028-make-medreg-import-pass

Summary

  • Make medreg imports pass

Commits

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!

---

Make medreg imports pass

Yesterday importing medreg doctors failed with a timeout after 634 doctors. Must find a way to make it more resilient. Should have the following characterstics

  • Possible to restart a failed job at the failing id
  • Retry an import after a timeout (e.g. of 10 seconds)

Started a helper class ResilientLoop in util/resilient_loop.rb with the corresponding unit test. Done. Looks nice for me.

Restarting jobs/import_regmed_doctors and waiting for errors or completion. Cannot run import for regmed-companies at the same time as it is also an Updater job and a duplicate PID is detected.

Pushed commits Getting doctors via GLN works and Added resilient loop of branch medreg

Import of doctors is running. In bin/admin I discovered that somehow the doctors-array was deleted 

doctors.size
-> 830
ch.oddb> doctor_count
-> 34663
ch.oddb> recount
-> #<Thread:0x0000000a653538>
<.. after a few minutes>
ch.oddb> doctor_count
-> 833

I have no idea of why. At the end I must restart the import, reload the database and check whether I really delete all doctors (don't know where and why). Also I should have started importing the companies as there are only about 3000 of them (taking 2 or 3 seconds each) compared to the 5 seconds for over 33'000 doctors.

I think I have an error when creating the doctor's address. I must define an addres_type. Must check the following items (from attr_accessor)

  • :name
  • :additional_lines
  • :address
  • :location
  • :title
  • :fon (done)
  • :fax (done)
  • :canton
  • :type

Found not correctly defined method refactor_addresses in oddbapp. Should probably be deleted.

Stopping import after 1000 doctors to fix problems with address. Drop/restore database. Will run import of 7601000648372 which is not present in ch.oddb.org (exam 2013)

Now I have a problem (recursive locking) 

2014-10-28 15:46:17 +0100:  MedregDoctorPlugin parsing /var/www/oddb.org/data/xls/doctors_latest.xlsx
2014-10-28 15:47:21 +0100:  MedregDoctorPlugin get_detail_to_glns 1. first 10 are ["7601000648372"]
2014-10-28 15:47:21 +0100:  MedregDoctorPlugin Searching for company with GLN 7601000648372 (1/1)
2014-10-28 15:47:26 +0100:  MedregDoctorPlugin {:ean13=>"7601000648372", :name=>"Tebyanian Khosropanahi", :firstname=>"Mojdeh", :exam=>"Ärztin/Arzt", :specialities=>["            Praktische Ärztin _ Praktischer Arzt,2012,Schweiz"], :addresses=>[#<ODDB::Address2:0x000000094db378 @revision=2014-10-28 15:47:26 +0100, @odba_id=32356661, @oid=32356661, @additional_lines=["Chemin des Uttins 4", "2013 Colombier"], @fon="032 841 33 33", @fax="032 841 33 32", @type="at_praxis", @canton="Neuenburg">, #<ODDB::Address2:0x0000000bf8bd20 @revision=2014-10-28 15:47:26 +0100, @odba_id=32356662, @oid=32356662, @additional_lines=["Chemin des Uttins  4", "2013 Colombier"], @fon="032 841 33 33", @fax=[], @type="at_praxis", @canton="Neuenburg">]}
2014-10-28 15:47:31 +0100:  MedregDoctorPlugin Searching for company with GLN 7601000648372 (2/1)
2014-10-28 15:47:36 +0100:  MedregDoctorPlugin {:ean13=>"7601000648372", :name=>"Tebyanian Khosropanahi", :firstname=>"Mojdeh", :exam=>"Ärztin/Arzt", :specialities=>["            Praktische Ärztin _ Praktischer Arzt,2012,Schweiz"], :addresses=>[#<ODDB::Address2:0x00000007716d38 @revision=2014-10-28 15:47:36 +0100, @odba_id=32356663, @oid=32356663, @additional_lines=["Chemin des Uttins 4", "2013 Colombier"], @fon="032 841 33 33", @fax="032 841 33 32", @type="at_praxis", @canton="Neuenburg">, #<ODDB::Address2:0x000000077353a0 @revision=2014-10-28 15:47:36 +0100, @odba_id=32356664, @oid=32356664, @additional_lines=["Chemin des Uttins  4", "2013 Colombier"], @fon="032 841 33 33", @fax=[], @type="at_praxis", @canton="Neuenburg">]}
2014-10-28 15:47:41 +0100:  MedregDoctorPlugin Searching for company with GLN 7601000648372 (3/1)
2014-10-28 15:47:46 +0100:  MedregDoctorPlugin {:ean13=>"7601000648372", :name=>"Tebyanian Khosropanahi", :firstname=>"Mojdeh", :exam=>"Ärztin/Arzt", :specialities=>["            Praktische Ärztin _ Praktischer Arzt,2012,Schweiz"], :addresses=>[#<ODDB::Address2:0x0000000c82d0f0 @revision=2014-10-28 15:47:46 +0100, @odba_id=32356665, @oid=32356665, @additional_lines=["Chemin des Uttins 4", "2013 Colombier"], @fon="032 841 33 33", @fax="032 841 33 32", @type="at_praxis", @canton="Neuenburg">, #<ODDB::Address2:0x0000000c82b688 @revision=2014-10-28 15:47:46 +0100, @odba_id=32356666, @oid=32356666, @additional_lines=["Chemin des Uttins  4", "2013 Colombier"], @fon="032 841 33 33", @fax=[], @type="at_praxis", @canton="Neuenburg">]}
/var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: doctors (Medreg) - 10/2014 size Plugin: ODDB::Doctors::MedregDoctorPlugin
Error: ThreadError
Message: deadlock; recursive locking
Backtrace:
<internal:prelude>:8:in `lock'
<internal:prelude>:8:in `synchronize'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:31:in `ensure in next_connection'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:31:in `next_connection'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/connection_pool.rb:38:in `method_missing'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:508:in `restore_collection'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:236:in `fetch_collection'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:632:in `restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:318:in `block in fetch_or_restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:317:in `fetch_or_restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:65:in `block in bulk_restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:62:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:62:in `bulk_restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:56:in `bulk_fetch'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:260:in `fetch_collection'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:632:in `restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:318:in `block in fetch_or_restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:317:in `fetch_or_restore'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:640:in `restore_object'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:605:in `load_object'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:226:in `block in fetch'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:313:in `fetch_or_do'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:225:in `fetch'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:49:in `odba_receiver'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/util/oddbapp.rb:619:in `doctor_by_gln'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:266:in `store_doctor'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:193:in `block (2 levels) in get_detail_to_glns'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `call'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run'
/usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:118:in `block in get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:116:in `each'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:116:in `get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:91:in `update'
/var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors'
jobs/import_regmed_doctors:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_regmed_doctors:12:in `<module:Util>'
jobs/import_regmed_doctors:11:in `<module:ODDB>'
jobs/import_regmed_doctors:10:in `<main>'
/var/www/oddb.org/log/job.pid is deleted

After increasing the timeout to 120 seconds (measured time was 46) the error when away. But the new doctor was not correctly stored in app.doctors. Adding a new call to app.doctors.odba_store at the end of the import.

Fixed this problem and searching via bin/admin works 

ch.oddb> doctor_by_gln('7601000648372').name
-> Tebyanian Khosropanahi
doctor_by_gln('7601000648372').firstname
-> Mojdeh
ch.oddb> doctor_by_gln('7601000648372').addresses.first
-> #<ODDB::Address2:0x007fd9a13c9308>
ch.oddb> doctor_by_gln('7601000648372').addresses.first.fon
-> 032 841 33 33
ch.oddb> doctor_by_gln('7601000648372').addresses.first.fax
-> 032 841 33 32
ch.oddb> doctor_by_gln('7601000648372').addresses.first.additional_lines
-> ["Chemin des Uttins 4", "2013 Colombier"]
ch.oddb> doctor_by_gln('7601000648372').addresses.first.type
-> at_praxis

but http://oddb-ci2.dyndns.org/de/gcc/doctor/ean/7601000648372 shows Die von Ihnen gewünschte Information ist leider nicht mehr vorhanden. Searching in home_doctors via Tebyanian or Mojdeh did not show any results neither.

Now running jobs/import_regmed_companies 7601002023177. Could not verify inside bin/admin whether the new company was created correctly, but the companies.size had increased by 1. Starting importing all companies to watch whether we can import them all. Will fix the error later.

Got the following error 

2014-10-28 18:17:04 +0100:  MedregCompanyPlugin {:ean13=>"7601001017320", :name=>"Pharmacie de Leytron", :plz=>"1912", :location=>"Leytron", :address=>["Route de Saillon 6"], :typ=>"öffentliche Apotheke"}
failsafe rescued DBI::ProgrammingError < StandardError
message contents do not agree with length in message type "C"
lost synchronization with server: got message type "I", length 1314080082
/usr/local/lib/ruby/site_ruby/1.9.1/dbd/pg/statement.rb:62:in `rescue in execute'
/usr/local/lib/ruby/site_ruby/1.9.1/dbd/pg/statement.rb:37:in `execute'
/usr/local/lib/ruby/site_ruby/1.9.1/dbi/handles/statement.rb:116:in `execute'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:263:in `block in ensure_object_connections'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:262:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb:262:in `ensure_object_connections'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:553:in `store_object_connections'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:512:in `store'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:274:in `odba_isolated_store'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:395:in `block in odba_store_unsaved'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:392:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:392:in `odba_store_unsaved'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:380:in `odba_store'
/var/www/oddb.org/src/util/persistence.rb:226:in `issue_create'
/var/www/oddb.org/src/util/oddbapp.rb:110:in `block in create'
/var/www/oddb.org/src/util/failsafe.rb:10:in `call'
/var/www/oddb.org/src/util/failsafe.rb:10:in `failsafe'
/var/www/oddb.org/src/util/oddbapp.rb:109:in `create'
/var/www/oddb.org/src/util/persistence.rb:288:in `block in resolve'
/var/www/oddb.org/src/util/persistence.rb:278:in `each'
/var/www/oddb.org/src/util/persistence.rb:278:in `resolve'
/var/www/oddb.org/src/util/persistence.rb:256:in `issue_update'
/var/www/oddb.org/src/util/oddbapp.rb:131:in `block in update'
/var/www/oddb.org/src/util/failsafe.rb:10:in `call'
/var/www/oddb.org/src/util/failsafe.rb:10:in `failsafe'
/var/www/oddb.org/src/util/oddbapp.rb:129:in `update'
/var/www/oddb.org/src/util/oddbapp.rb:1533:in `update'
/var/www/oddb.org/src/plugin/medreg_company.rb:214:in `store_company'
/var/www/oddb.org/src/plugin/medreg_company.rb:159:in `block (2 levels) in get_detail_to_glns'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `call'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run'
/usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run'
/var/www/oddb.org/src/plugin/medreg_company.rb:118:in `block in get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_company.rb:116:in `each'
/var/www/oddb.org/src/plugin/medreg_company.rb:116:in `get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_company.rb:93:in `update'
/var/www/oddb.org/src/util/updater.rb:249:in `block in update_medreg_companies'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:247:in `update_medreg_companies'
jobs/import_regmed_companies:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_regmed_companies:12:in `<module:Util>'
jobs/import_regmed_companies:11:in `<module:ODDB>'
jobs/import_regmed_companies:10:in `<main>'

The import_doctors got a timeout error (after 3 retries) 

2014-10-28 18:43:11 CET MedregDoctorPlugin {:ean13=>"7601000001122", :name=>"Canova", :firstname=>"Gérard", :exam=>"Ärztin/Arzt", :specialities=>["            Allgemeine Innere Medizin,2002,Schweiz"], :addresses=>[#<ODDB::Address2:0x0000000bfb7628 @revision=2014-10-28 18:43:11 +0100, @odba_id=32356739, @oid=32356739, @additional_lines=["Avenue de la Gare 2A", "1003 Lausanne"], @fon="021/312.19.31", @fax="021/312.79.39", @type="at_praxis", @canton="Waadt">]}
2014-10-28 18:43:12 CET MedregDoctorPlugin store_doctor 7601000001122 update in database. pointer :!doctor,25216.. Have now 34663 doctors. hash {:ean13=>"7601000001122", :exam=>0, :firstname=>"Gérard", :name=>"Canova", :specialities=>["[\"            Allgemeine Innere Medizin,2002,Schweiz\"]"], :addresses=>[#<ODDB::Address2:0x0000000bfb7628 @revision=2014-10-28 18:43:11 +0100, @odba_id=32356739, @oid=32356739, @additional_lines=["Avenue de la Gare 2A", "1003 Lausanne"], @fon="021/312.19.31", @fax="021/312.79.39", @type="at_praxis", @canton="Waadt">]}
2014-10-28 18:43:12 CET MedregDoctorPlugin Searching for company with GLN 7601000001139 (33/32953)
2014-10-28 18:43:16 CET MedregDoctorPlugin {:ean13=>"7601000001139", :name=>"Clerc", :firstname=>"Nicole", :exam=>"Ärztin/Arzt", :specialities=>["            Psychiatrie und Psychotherapie,1996,Schweiz"], :addresses=>[#<ODDB::Address2:0x000000057e6b70 @revision=2014-10-28 18:43:16 +0100, @odba_id=32356742, @oid=32356742, @additional_lines=["Route Haldimand 14", "1003 Lausanne"], @fon=[], @fax=[], @type="at_praxis", @canton="Waadt">]}
2014-10-28 18:43:16 CET MedregDoctorPlugin store_doctor ptr :!doctor. pointer :!create,:!doctor..
2014-10-28 18:45:12 CET MedregDoctorPlugin Searching for company with GLN 7601000001139 (34/32953)
2014-10-28 18:45:16 CET MedregDoctorPlugin {:ean13=>"7601000001139", :name=>"Clerc", :firstname=>"Nicole", :exam=>"Ärztin/Arzt", :specialities=>["            Psychiatrie und Psychotherapie,1996,Schweiz"], :addresses=>[#<ODDB::Address2:0x0000002802b890 @revision=2014-10-28 18:45:16 +0100, @odba_id=32356744, @oid=32356744, @additional_lines=["Route Haldimand 14", "1003 Lausanne"], @fon=[], @fax=[], @type="at_praxis", @canton="Waadt">]}
2014-10-28 18:45:16 CET MedregDoctorPlugin store_doctor ptr :!doctor. pointer :!create,:!doctor..
2014-10-28 18:47:12 CET MedregDoctorPlugin Searching for company with GLN 7601000001139 (35/32953)
2014-10-28 18:47:16 CET MedregDoctorPlugin {:ean13=>"7601000001139", :name=>"Clerc", :firstname=>"Nicole", :exam=>"Ärztin/Arzt", :specialities=>["            Psychiatrie und Psychotherapie,1996,Schweiz"], :addresses=>[#<ODDB::Address2:0x0000000b7af070 @revision=2014-10-28 18:47:16 +0100, @odba_id=32356746, @oid=32356746, @additional_lines=["Route Haldimand 14", "1003 Lausanne"], @fon=[], @fax=[], @type="at_praxis", @canton="Waadt">]}
2014-10-28 18:58:59 CET /var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
2014-10-28 18:58:59 CET Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: doctors (Medreg) - 10/2014 size Plugin: ODDB::Doctors::MedregDoctorPlugin
Error: Timeout::Error
Message: Timeout::Error
Backtrace:
/var/www/oddb.org/src/util/resilient_loop.rb:68:in `rescue in try_run'
/var/www/oddb.org/src/util/resilient_loop.rb:61:in `try_run'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:117:in `block in get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:115:in `each'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:115:in `get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:90:in `update'
/var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors'
jobs/import_regmed_doctors:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_regmed_doctors:12:in `<module:Util>'
jobs/import_regmed_doctors:11:in `<module:ODDB>'
jobs/import_regmed_doctors:10:in `<main>'
view · edit · sidebar · attach · print · history
Page last modified on October 29, 2014, at 08:23 AM