---
Checking why the import failed. Corrected some minor details in medreg_company, made it look even more similar to medreg_doctor. This improves logging. Restarting import_companies. Downloading person/companies-xls file failed again today.
Running the companies import fails after 30, then 12 companies and shows Personensuche-Resultat Die Personensuche dauerte zu lange. Bitte versuchen Sie es später noch einmal. on https://www.medreg.admin.ch/MedReg/PersonenSucheResultat.aspx. This persists over several minutes.
In resilient_loop fixed comparing using to_s.eql? instead of ==. Restarted medreg_doctors now restarts from the ID given.
Must catch and handle correctly the following error
Error: Mechanize::ResponseCodeError Message: 503 => Net::HTTPServiceUnavailable for http://www.medregom.admin.ch/de/Detail/Detail?pid=31002 -- unhandled response Backtrace: /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:304:in `fetch' /usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:407:in `get' /var/www/oddb.org/src/plugin/medreg_doctor.rb:165:in `block (2 levels) in get_detail_to_glns' /var/www/oddb.org/src/util/resilient_loop.rb:63:in `call' /var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run' /usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout' /usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout' /var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run' /var/www/oddb.org/src/plugin/medreg_doctor.rb:118:in `block in get_detail_to_glns' /var/www/oddb.org/src/plugin/medreg_doctor.rb:113:in `each' /var/www/oddb.org/src/plugin/medreg_doctor.rb:113:in `get_detail_to_glns' /var/www/oddb.org/src/plugin/medreg_doctor.rb:90:in `update' /var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors' /var/www/oddb.org/src/util/updater.rb:526:in `call' /var/www/oddb.org/src/util/updater.rb:526:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors' jobs/import_regmed_doctors:14:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_regmed_doctors:12:in `<module:Util>' jobs/import_regmed_doctors:11:in `<module:ODDB>' jobs/import_regmed_doctors:10:in `<main>'
Strategy: Wait some time, then restart. This seems to work.
But got now the following error
Error: NoMethodError Message: undefined method `[]' for nil:NilClass Backtrace: /var/www/oddb.org/src/plugin/medreg_doctor.rb:186:in `block (2 levels) in get_one_doctor' /var/www/oddb.org/src/plugin/medreg_doctor.rb:181:in `upto' /var/www/oddb.org/src/plugin/medreg_doctor.rb:181:in `each' /var/www/oddb.org/src/plugin/medreg_doctor.rb:181:in `block in get_one_doctor' /var/www/oddb.org/src/util/resilient_loop.rb:63:in `call' /var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run' /usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout' /usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout' /var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run' /var/www/oddb.org/src/plugin/medreg_doctor.rb:110:in `get_one_doctor' /var/www/oddb.org/src/plugin/medreg_doctor.rb:211:in `block in get_detail_to_glns' /var/www/oddb.org/src/plugin/medreg_doctor.rb:202:in `each' /var/www/oddb.org/src/plugin/medreg_doctor.rb:202:in `get_detail_to_glns' /var/www/oddb.org/src/plugin/medreg_doctor.rb:90:in `update' /var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors' /var/www/oddb.org/src/util/updater.rb:526:in `call' /var/www/oddb.org/src/util/updater.rb:526:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors' jobs/import_regmed_doctors:14:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_regmed_doctors:12:in `<module:Util>' jobs/import_regmed_doctors:11:in `<module:ODDB>'
This is a rare case. URL is http://www.medregom.admin.ch/de/Suche/Detail/?gln=7601000010735&vorname=Philippe+Marc&name=Cevey. He is a medical doctor and a dentist and in this case my search algorithm produced wrong result. Made a quick fix, but for this case I must add unit test to ensure that everything is okay.
Now my import hangs after
2014-10-29 11:16:57 +0100: MedregDoctorPlugin {:ean13=>"7601000011329", :name=>"Ona", :firstname=>"Marie Belen", :exam=>"Ärztin/Arzt", :specialities=>["Dermatologie und Venerologie,1996,Schweiz"], :experiences=>[], :addresses=>[#<ODDB::Address2:0x00000008242b30 @revision=2014-10-29 11:16:57 +0100, @odba_id=32358120, @oid=32358120, @additional_lines=["Grand-Chêne 4", "1003 Lausanne"], @fon="021/320.39.00", @fax="021/320.39.05", @type="at_praxis", @canton="Waadt">]}
with almost no CPU load after sending a kill, the CPU load got upto 200 % (for several minutes). Sending a kill -9. Restarting postgresql and import job.
Fixed parsing 7601000010735 (dentist/medical doctor). Restarting import.
Tasks to be done are:
BTM Berechtigung, Bewilligung Selbstdispensation
Betriebstyp, BTM Berechtigung
:business_area = [nil, 'ba_hospital', 'ba_pharma', 'ba_insurance', 'ba_doctor', 'ba_health', 'ba_info' ],
We found the following values (+ my proposal)
Pushed commit Fix creating doctors, companies and addresses.
Got the following error
Plugin: ODDB::Doctors::MedregDoctorPlugin Error: NoMethodError Message: undefined method `ean13' for nil:NilClass Backtrace: /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/util/oddbapp.rb:620:in `block in doctor_by_gln' /var/www/oddb.org/src/util/oddbapp.rb:619:in `each' /var/www/oddb.org/src/util/oddbapp.rb:619:in `doctor_by_gln' /var/www/oddb.org/src/plugin/medreg_doctor.rb:305:in `store_doctor' /var/www/oddb.org/src/plugin/medreg_doctor.rb:206:in `block in get_one_doctor' /var/www/oddb.org/src/util/resilient_loop.rb:63:in `call' /var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run' /usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout' /usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout' /var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run' /var/www/oddb.org/src/plugin/medreg_doctor.rb:152:in `get_one_doctor' /var/www/oddb.org/src/plugin/medreg_doctor.rb:224:in `block in get_detail_to_glns' /var/www/oddb.org/src/plugin/medreg_doctor.rb:214:in `each' /var/www/oddb.org/src/plugin/medreg_doctor.rb:214:in `get_detail_to_glns' /var/www/oddb.org/src/plugin/medreg_doctor.rb:90:in `update' /var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors' /var/www/oddb.org/src/util/updater.rb:526:in `call' /var/www/oddb.org/src/util/updater.rb:526:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors' jobs/import_regmed_doctors:14:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_regmed_doctors:12:in `<module:Util>' jobs/import_regmed_doctors:11:in `<module:ODDB>' jobs/import_regmed_doctors:10:in `<main>'
I think this is due to the fact that I had another problem while importing this doctor. Therefore deleting it via bin/admin and restarting the import.
-> $x = nil; doctors.values.each{ |x| if x.ean13 == nil then $x = x; break ; end}
ch.oddb> doctors.delete($x.oid)
-> #<ODDB::Doctor:0x0000000bbbc730>
ch.oddb> doctors.odba_isolated_store
-> Hash
Made import of companies loop if medreg response with searching takes too long. Restarted import of companies. This time the import completes. Here the end of the log
2014-10-29 18:41:07 +0100: MedregCompanyPlugin store_company 7610840000059 update in database. pointer :!create,:!company,32368545... Have now 3753 companies. hash {:ean13=>"7610840000059", :name=>nil, :addresses=>[#<ODDB::Address2:0x0000000b8b2198 @revision=2014-10-29 18:41:06 +0100, @odba_id=32368543, @oid=32368543, @additional_lines=[], @fon=[], @fax=[], @address=["Rue de Saugy 1"], @location="1023 Crissier">]}
2014-10-29 18:41:07 +0100: MedregCompanyPlugin Searching for company with GLN Apotheke & Drogerie Ruopigen AG (2991/2998).
2014-10-29 18:41:07 +0100: MedregCompanyPlugin could not find gln Apotheke & Drogerie Ruopigen AG
2014-10-29 18:41:07 +0100: MedregCompanyPlugin Searching for company with GLN Farmacia Chimica Internazionale (2992/2998).
2014-10-29 18:41:07 +0100: MedregCompanyPlugin could not find gln Farmacia Chimica Internazionale
2014-10-29 18:41:07 +0100: MedregCompanyPlugin Searching for company with GLN Farmacia Sun Store (2993/2998).
2014-10-29 18:41:08 +0100: MedregCompanyPlugin could not find gln Farmacia Sun Store
2014-10-29 18:41:08 +0100: MedregCompanyPlugin Searching for company with GLN MV SANTE Beaumont SA (2994/2998).
2014-10-29 18:41:08 +0100: MedregCompanyPlugin could not find gln MV SANTE Beaumont SA
2014-10-29 18:41:08 +0100: MedregCompanyPlugin Searching for company with GLN Mathilde Escher-Heim (2995/2998).
2014-10-29 18:41:09 +0100: MedregCompanyPlugin could not find gln Mathilde Escher-Heim
2014-10-29 18:41:09 +0100: MedregCompanyPlugin Searching for company with GLN Pharmacie Amavita des Deux-Ponts (2996/2998).
2014-10-29 18:41:09 +0100: MedregCompanyPlugin could not find gln Pharmacie Amavita des Deux-Ponts
2014-10-29 18:41:09 +0100: MedregCompanyPlugin Searching for company with GLN Pharmacie Solidarité (2997/2998).
2014-10-29 18:41:09 +0100: MedregCompanyPlugin could not find gln Pharmacie Solidarité
2014-10-29 18:41:09 +0100: MedregCompanyPlugin Searching for company with GLN Pharmacies BENU SA (2998/2998).
2014-10-29 18:41:10 +0100: MedregCompanyPlugin could not find gln Pharmacies BENU SA
2014-10-29 18:41:11 +0100: MedregCompanyPlugin Saved 2978 companies in /var/www/oddb.org/data/txt/companies_2014.10.29.yaml
/var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - companies (Medreg) - 10/2014 size Companies update
Number of companies: 3753
New companies: 2969
Updated companies: 10
Deleted companies: 0
/var/www/oddb.org/log/job.pid is deleted
TODO: Add entries like could not find gln Mathilde Escher-Heim to the error report. But inside the XLSX file there really are 8 entries without a GLN.