Index>

20141027-parse-medreg-companies

Summary

• parse medreg pages companies
• parse medregom pages doctors

Commits

Index

• parse medreg pages companies
• parse medregom pages doctors

Keep in Mind

• Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
• I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
• The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!

---

parse medregom pages doctors

Trying to use mechanize for parsing the medregom pages. As http://www.medregom.admin.ch/ seems to work fine now, it was easy to get the correct response via mechanize for my example useage. Adapting parsing the output and making the unit test pass.

Unit-tests pass now. We don't handle multiple permissions for different cantons. Running jobs/import_regmed_doctors on oddb-ci2 and fixing the errors when trying to add a new doctor.

As the URL https://www.medregbm.admin.ch/Publikation/CreateExcelListMedizinalPersons suddenly ceased to work I changed the implementation of get_latest_file to return true, path when the daily xls is present, but the latest file is missing. Now the import starts working and tries to import 32953 doctors.

Had to restart the import after 200 entries, as I did not handle correctly some situation. Added log output for them. Okay: Problem is Nationaltät: Österreich, where the Ö does not match rubies \w. Now already imported 294 doctors. Will look closer at the import tomorrow morning.

Now trying to implement a timeout/recovery strategy when importing the doctors or the companies.

parse medreg pages companies

When running jobs/import_regmed_companies it did not complete. It just stopped after 220 to 260 companies. Once with a Ruby error.

Will rework the importer adding a timeout for each import and saving the position of the last imported GLN in a file to be able to restart the import job.

As importing two GLNs worked, starting import with all doctors now.