view · edit · sidebar · attach · print · history

Index>

20141113-medreg-import-and-view

Summary

  • Make medreg import pass
  • Add view for doctors/hospital/pharamacies

Commits

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!

---

Add view for doctors/hospital/pharamacies

Create a new class hc_provider (health care provider) which collects data from the 3 relevant classes.

Added a lot of stuff for hc_providers (But still missing some pieces). Added some unit tests to ensure that everythings compiles okay. See commit Initial setup of hc_providers

The following files were added 

[medreg a27c80f] Initial setup of hc_providers
 24 files changed, 1081 insertions(+), 182 deletions(-)
 create mode 100644 src/state/hc_providers/global.rb
 create mode 100644 src/state/hc_providers/hc_provider.rb
 create mode 100644 src/state/hc_providers/hc_providerlist.rb
 create mode 100644 src/state/hc_providers/init.rb
 create mode 100644 src/state/hc_providers/limit.rb
 create mode 100644 src/state/hc_providers/setpass.rb
 create mode 100644 src/state/hc_providers/vcard.rb
 create mode 100644 src/view/hc_providers/centeredsearchform.rb
 create mode 100644 src/view/hc_providers/hc_provider.rb
 create mode 100644 src/view/hc_providers/hc_providerlist.rb
 create mode 100644 src/view/hc_providers/search.rb
 create mode 100644 src/view/hc_providers/vcard.rb
 create mode 100644 src/view/hc_providers/welcomehead.rb
 create mode 100644 test/test_model/hc_provider.rb
 create mode 100644 test/test_state/hc_providers/hc_provider.rb
 create mode 100644 test/test_view/hc_providers/hc_provider.rb

Make medreg import pass

I started a Medreg-import at 18:27. Around 19:50 it took 4 retries to import 7601000032546 and 22:25 2 retries for 7601000060426.

It aborted at 23:37 after 3212 doctors when importing doctor with 7601000078261 

Error: NoMethodError
Message: undefined method `[]' for nil:NilClass
Backtrace:
/var/www/oddb.org/src/plugin/medreg_doctor.rb:128:in `block in parse_details'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:124:in `upto'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:124:in `each'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:124:in `parse_details'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:204:in `block in get_one_doctor'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `call'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run'
/usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:152:in `get_one_doctor'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:225:in `block in get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:215:in `each'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:215:in `get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:90:in `update'
/var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors'
jobs/import_regmed_doctors:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_regmed_doctors:12:in `<module:Util>'
jobs/import_regmed_doctors:11:in `<module:ODDB>'
jobs/import_regmed_doctors:10:in `<main>'

Added GLN 7601000078261 to my unit tests and it fails at the same ruby code. Found the reason. Was the first additional qualification from austria which must be matched using [Ö\w]+ and not only \w+. Fixed this problem with commit Added fix for doctor with austrian additional qualification

Corrected log message while importing medreg doctors with commit Better statistic at importing medreg doctors (skip_to)

At 13:06 http://www.medregom.admin.ch/ reported again Server Error in '/' Application. Exception of type 'System.OutOfMemoryException' was thrown.

Did not catch correctly a timeout error, as I got the following mail 

Error: Net::HTTP::Persistent::Error
Message: too many connection resets (due to Timeout::Error - Timeout::Error) after 3339 requests on 100323920, last used 60.061726084 seconds ago
Backtrace:
/usr/local/lib/ruby/1.9.1/net/protocol.rb:146:in `rescue in rbuf_fill'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:140:in `rbuf_fill'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
/usr/local/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
/usr/local/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
/usr/local/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
/usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
/usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
/usr/local/lib/ruby/1.9.1/net/http.rb:1293:in `request'
/usr/local/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:999:in `request'
/usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:258:in `fetch'
/usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:1229:in `post_form'
/usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:469:in `post'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:191:in `block in get_one_doctor'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `call'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run'
/usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:162:in `get_one_doctor'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:237:in `block in get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:226:in `each'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:226:in `get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:91:in `update'
/var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors'
jobs/import_regmed_doctors:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_regmed_doctors:12:in `<module:Util>'
jobs/import_regmed_doctors:11:in `<module:ODDB>'
jobs/import_regmed_doctors:10:in `<main>'

Fixed with commit Catch timeout errors in medreg_doctor import

Will add HH.MM to the timestamp of the generate yaml file to avoid overriding its content if the import is started again.

Also must (as a fallback) use the latest downloaded xlsx file if the download failed and report this in the generated mail.

Must also look once at errors like this 

RuntimeError: can't modify frozen String when updating index 'doctor_index' with a ODDB::Doctor
["/var/www/oddb.org/src/util/searchterms.rb:91:in `force_encoding'", "/var/www/oddb.org/src/util/searchterms.rb:91:in `block in search_terms'", "/var/www/oddb.org/src/util/searchterms.rb:87:in `each'", "/var/www/oddb.org/src/util/searchterms.rb:87:in `inject'"]
[...]

Not handling correctly some veterinary doctors 

2014-11-13 16:32:35 +0100:  MedregDoctorPlugin GLN: 7601000332851 no match for line 
            FZ Identifikation von Pferden / Pferdepass

            Schweiz

Got another error 

2014-11-13 19:32:03 +0100:  MedregDoctorPlugin store_doctor 7601000383440 create in database. pointer :!create,:!doctor,32404959... Have now 38718 doctors. hash {:ean13=>"7601000383440", :exam=>0, :firstname=>"Roger", :name=>"Fitzi", :specialities=>["[]"], :addresses=>[#<ODDB::Address2:0x000000215d1878 @revision=2014-11-13 19:32:00 +0100, @odba_id=32404958, @oid=32404958, @additional_lines=["Zürcherstrasse 234", "9014 St. Gallen"], @fon="071 311 21 22", @fax="071 311 12 76", @type="at_praxis", @canton="St. Gallen">]}
2014-11-13 19:32:03 +0100:  MedregDoctorPlugin Searching for doctor with GLN 7601000383457. (at 11485 of 26373 to import of 33023).
2014-11-13 19:32:06 +0100:  MedregDoctorPlugin Start saving @app.doctors.odba_store
2014-11-13 19:32:08 +0100:  MedregDoctorPlugin Finished @app.doctors.odba_store
2014-11-13 19:32:14 +0100:  MedregDoctorPlugin Saved 11347 doctors in /var/www/oddb.org/data/txt/doctors_2014.11.12-1850.yaml
/var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: doctors (Medreg) - 11/2014 size 2510 with 0 attachments. Plugin: ODDB::Doctors::MedregDoctorPlugin
Error: Net::HTTP::Persistent::Error
Message: too many connection resets (due to Connection reset by peer - Errno::ECONNRESET) after 3366 requests on 104469520, last used 0.484677055 seconds ago
Backtrace:
/usr/local/lib/ruby/1.9.1/net/protocol.rb:141:in `read_nonblock'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:141:in `rbuf_fill'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:122:in `readuntil'
/usr/local/lib/ruby/1.9.1/net/protocol.rb:132:in `readline'
/usr/local/lib/ruby/1.9.1/net/http.rb:2562:in `read_status_line'
/usr/local/lib/ruby/1.9.1/net/http.rb:2551:in `read_new'
/usr/local/lib/ruby/1.9.1/net/http.rb:1319:in `block in transport_request'
/usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `catch'
/usr/local/lib/ruby/1.9.1/net/http.rb:1316:in `transport_request'
/usr/local/lib/ruby/1.9.1/net/http.rb:1293:in `request'
/usr/local/lib/ruby/gems/1.9.1/gems/net-http-persistent-2.9.4/lib/net/http/persistent.rb:999:in `request'
/usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:258:in `fetch'
/usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:1229:in `post_form'
/usr/local/lib/ruby/gems/1.9.1/gems/mechanize-2.5.1/lib/mechanize.rb:469:in `post'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:190:in `block in get_one_doctor'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `call'
/var/www/oddb.org/src/util/resilient_loop.rb:63:in `block in try_run'
/usr/local/lib/ruby/1.9.1/timeout.rb:68:in `timeout'
/usr/local/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/var/www/oddb.org/src/util/resilient_loop.rb:62:in `try_run'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:161:in `get_one_doctor'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:236:in `block in get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:225:in `each'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:225:in `get_detail_to_glns'
/var/www/oddb.org/src/plugin/medreg_doctor.rb:91:in `update'
/var/www/oddb.org/src/util/updater.rb:237:in `block in update_medreg_doctors'
/var/www/oddb.org/src/util/updater.rb:526:in `call'
/var/www/oddb.org/src/util/updater.rb:526:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:235:in `update_medreg_doctors'
jobs/import_regmed_doctors:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_regmed_doctors:12:in `<module:Util>'
jobs/import_regmed_doctors:11:in `<module:ODDB>'
jobs/import_regmed_doctors:10:in `<main>'
/var/www/oddb.org/log/job.pid is deleted
view · edit · sidebar · attach · print · history
Page last modified on November 13, 2014, at 08:08 PM