During the night I ran the importer for doctors twice (removing the latest file between). This did not work as expected as in the second run the new address is not correctly recognisec, eg. I found lines like
2017-12-06 00:35:49 +0100: RefdataDoctorPlugin store_doctor update 7601000108951 oid 37292309 in database. pointer nil {"addresses"=>"no new address"}
This must be fixed.
This happens if I find an entry in the refdatabase
<ITEM xmlns="http://refdatabase.refdata.ch/Partner_out" DT="2016-01-06T00:00:00">
<PTYPE>NAT</PTYPE>
<GLN>7601000108951</GLN>
<STATUS>A</STATUS>
<STDATE>2006-01-01T00:00:00</STDATE>
<LANG>DE</LANG>
<DESCR1>Ackermann</DESCR1>
<DESCR2>Hans-Peter</DESCR2>
<ROLE>
<TYPE>DoctMed</TYPE>
</ROLE>
</ITEM>
where only the search via https://www.medreg.admin.ch returns an addresss. I think in this case I should not report any change. Fixing this problem.
Also remarked that the hospital pharmacies are not correctly update (e.g, 7601002028127 Bethesda-Spital) always appears in the list of new partners.
Renaming jobs, files, etc from doctors -> nat and partners - jur. Done.
Now running the jobs/import_refdata_jur detects correctly hospital pharmacies, but does not correctly store new hospitals. Debugging with with pry.
When storing the hospitals I see the following inconsistency
[1] pry(#<ODDB::Companies::RefdataJurPlugin>)> @app.hospital_by_gln(7601002001229) => nil [2] pry(#<ODDB::Companies::RefdataJurPlugin>)> @app.hospitals[7601002001229].odba_id => 37533070 [3] pry(#<ODDB::Companies::RefdataJurPlugin>)>
This stems from the fact, that the ean13 was stored as a String and not as Integer. Whereas for companies ean13 is an Integer. Fixing create_hospital in src/util/oddbapp.rb
Pushed the following commits
Analysing the problem, why a search for Kriens does not list all pharmacies. bin/admin shows
-> Alters- und Pflegeheim Grossfeld
ch.oddb>
->
ch.oddb> pharmacy_by_gln(7601002051163).search_terms
-> Array
ch.oddb> pharmacy_by_gln(7601002051163).search_terms.find_all{|x| /kriens/i.match(x.downcase)}
-> ["6010 Kriens", "Kriens"]
ch.oddb> pharmacy_by_gln(7610840000042)
-> Coop Vitality Apotheke Kriens
ch.oddb> pharmacy_by_gln(7610840000042).search_terms
-> undefined method `split' for 7610840000042:Integer
After adding a unit-test and fixing this problems I must run sudo -u apache bundle-240 exec ruby-240 jobs/rebuild_indices company_index before I can see a result. It took about 10 minutes to rebuild the index, and now I can see the two new "Coop Vitality" pharmacies for Kriens.
Must get the link info for the details of each doctor on medreg. Parsing www.medreg.admin.ch using mechanize is not easy at all, as it uses a lot a javascript, which watir this is a simple job.
Pushed commits
Tried using the GLN to fetch the details, eg. https://www.medreg.admin.ch/MedReg/Summary.aspx?IdGln=7601000115690 fails with Message Der Parameter idPerson fehlt.
I do not like the idea to use watir, as this would require a running webbrowser and xvfb on the server. But may be there are other ruby gems we could use, eg.
Or does ext/swissreg/src/swissreg.rb handle a similar web site?
Also parsing should be a log faster. Whey is it so slow?
Runing the full watir tests showed, that I also have the same search term problems for doctors. Fixed with commit Fix search_terms for doctors
Running watir smoke_test before pushing commit Remove migel feedback
symptons were error messages like
Plugin: ODDB::RssPlugin Error: NoMethodError Message: undefined method `update' for #<ODDB::Doctor:0x007f22de5f4828> Backtrace: /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:591:in `block in update_indices' /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:590:in `each' /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:590:in `update_indices' /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:516:in `store' /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/persistable.rb:274:in `odba_isolated_store' /var/www/oddb.org/src/util/oddbapp.rb:1568:in `block in method_missing' /var/www/oddb.org/src/util/oddbapp.rb:1567:in `synchronize' /var/www/oddb.org/src/util/oddbapp.rb:1567:in `method_missing' /var/www/oddb.org/src/plugin/plugin.rb:108:in `update_rss_feeds' /var/www/oddb.org/src/plugin/rss.rb:175:in `update_swissmedic_feed' /var/www/oddb.org/src/plugin/rss.rb:194:in `update_recall_feed' /var/www/oddb.org/src/util/updater.rb:557:in `update_immediate_with_error_report' /var/www/oddb.org/src/util/updater.rb:411:in `update_recall_feed' /var/www/oddb.org/src/util/updater.rb:407:in `update_swissmedic_feeds' /var/www/oddb.org/src/util/updater.rb:210:in `run' jobs/import_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_daily:12:in `<module:Util>' jobs/import_daily:11:in `<module:ODDB>' jobs/import_daily:10:in `<main>'
Using pry sees
=> ["/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/stub.rb:112:in `method_missing'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:591:in `block in update_indices'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:590:in `each'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:590:in `update_indices'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:516:in `store'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/persistable.rb:274:in `odba_isolated_store'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/persistable.rb:395:in `block in odba_store_unsaved'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/persistable.rb:392:in `each'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/persistable.rb:392:in `odba_store_unsaved'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/persistable.rb:380:in `odba_store'", "/var/www/oddb.org/src/util/oddbapp.rb:1548:in `initialize'", "/var/www/oddb.org/config.ru:92:in `new'", "/var/www/oddb.org/config.ru:92:in `block in <main>'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:55:in `instance_eval'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:55:in `initialize'", "/var/www/oddb.org/config.ru:in `new'", "/var/www/oddb.org/config.ru:in `<main>'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:49:in `eval'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:49:in `new_from_string'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:40:in `parse_file'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/server.rb:319:in `build_app_and_options_from_config'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/server.rb:219:in `app'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/server.rb:354:in `wrapped_app'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/server.rb:283:in `start'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/server.rb:148:in `start'", "/var/www/oddb.org/vendor/ruby/2.4.0/gems/rack-2.0.3/bin/rackup:4:in `<top (required)>'", "/var/www/oddb.org/vendor/ruby/2.4.0/bin/rackup:22:in `load'", "/var/www/oddb.org/vendor/ruby/2.4.0/bin/rackup:22:in `<main>'"] [3] pry(#<ODDB::App>)>
As we have problems updating the index I will try to remove the dictionary: 'simple' from the doctor_index, as we have no such corresponding for companies.
Will drop the database and reload to try it. Removed the doctor_index from etc/index_definitions.yml. Now oddb.org starts up without any problem and bin/admin reports
ch.oddb> companies.size
-> 4635
ch.oddb> companies.first
-> [123, #<ODBA::Stub:47331105489240#508 @odba_class= @odba_container=47331118660100#3>]
ch.oddb> companies.values.first
-> athenstaedt AG
ch.oddb> doctors.size
-> 59762
ch.oddb> doctors.first
-> [8227, #<ODBA::Stub:47331137941240#425438 @odba_class= @odba_container=69925593816720#393389>]
ch.oddb> doctors.values.first
-> #<ODDB::Doctor:0x007f31a5923e60>
ch.oddb> doctors.values.first.ean13
-> 7601000216885
ch.oddb> doctors.values.first.ean13.class
-> String
ch.oddb> doctors.values.find_all{|x| x.ean13.is_a?(String)}.size
-> 26478
ch.oddb> doctors.values.find_all{|x| x.ean13.is_a?(Integer)}.size
-> 29224
No adding a new index (with a different name) for doctors, trying to rebuild it and restarting oddb.org afterwards. With the new index doctor_index_without_dictionary the services start okay. Rebuilding the index takes time (over 15 minutes). But know a search for doctors failes (tried Brunner, Schönbucher, Luzern).
Ran sudo -u apache bundle-240 exec ruby-240 jobs/rebuild_indices doctor_index on thinpower and this fixed the problem.