Index>

20141014-import-analysis

Summary

• Verify that import analysis works as expected
• Update list of medical doctors, hospitals and pharmacies

Commits

• https://github.com/ngiger/oddb.org/commit/f1fb94e503c8986d431bd8b278416f2677e3101a Added unit test for analysis
• https://github.com/ngiger/oddb.org/commit/2a83707369c35d7dbb7b0b2cc765780cfe49e716 Improved unit test for import analysis
• https://github.com/ngiger/oddb.org/commit/0f2a77dee07b8767d0969c82c13fc2b7161375fc Added logging for analysis import. Using transaction.

Index

• Verify that import analysis works as expected
• Update list of medical doctors, hospitals and pharmacies

Keep in Mind

• Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
• I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
• The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!

---

Update list of medical doctors, hospitals and pharmacies

Must use the following lists

• for doctors: https://www.medregbm.admin.ch/Publikation/CreateExcelListMedizinalPersons
• for hospital: https://www.medregbm.admin.ch/Publikation/CreateExcelListBetriebs

Zeno would like to combine the search for hospitals, doctors and drug store in a single view, accessible via Arzt/Spital/Apotheke When importing the data for the doctors it is important to know whether he/she is authorized to prescribe drugs. We should import all available fields.

Looking around the data a little bit one find curious entries like

7601000523174	Bay	Seraina Maria	8280	Kreuzlingen	Thurgau	CH	Ärztin/Arzt	Ja	Nein	
7601000523174	Bay	Seraina Maria	8280	Kreuzlingen	Thurgau	CH	Ärztin/Arzt	Ja	Ja	Verzicht
7601000079107	Zutter	Daniel Uriel	8702	Zollikon	Zürich	CH	Ärztin/Arzt	Ja	Nein
7601000079107	Zutter	Daniel Uriel	8588	Zihlschlacht	Thurgau	CH	Ärztin/Arzt	Ja	Nein
7601000079107	Zutter	Daniel Uriel	8580	Amriswil	Tessin	CH	Ärztin/Arzt	Ja	Nein
7601000019080	Zwingli	Martin	1110	Morges	Waadt	CH	Ärztin/Arzt	Ja	Nein
7601000019080	Zwingli	Martin	1110	Morges	Waadt	CH	Ärztin/Arzt	Ja	Nein
7601000019080	Zwingli	Martin	1007	Lausanne	Waadt	CH	Ärztin/Arzt	Ja	Nein
7601000019080	Zwingli	Martin	1110	Morges	Waadt	CH	Ärztin/Arzt	Ja	Nein
7601000019080	Zwingli	Martin	1007	Lausanne	Waadt	CH	Ärztin/Arzt	Ja	Nein
7601000019080	Zwingli	Martin	1000	Lausanne 7	Waadt	CH	Ärztin/Arzt	Ja	Nein

There are a few Betriebe without an EAN13

Altersheim Stollenweid		Stollenweid	2	8824	Schönenberg	Zürich	Schweiz	Spitalapotheke, Andere	6026 Verzeichnis a/b/c/f BetmVV-EDI
Farmacia Chimica Internazionale 	Farmacia Chimica Internazionale SA	via Balestra 	22B	6900	Lugano	Tessin	Schweiz	öffentliche Apotheke	6011 Verzeichnis a/b/c BetmVV-EDI
Farmacia Sun Store		Centro Ovale		6830	Chiasso	Tessin	Schweiz	öffentliche Apotheke	6011 Verzeichnis a/b/c BetmVV-EDI
Mathilde Escher-Heim		Lengghalde	1	8008	Zürich	Zürich	Schweiz	Spitalapotheke, Andere	6026 Verzeichnis a/b/c/f BetmVV-EDI
MV SANTE Beaumont SA		Avenue de Beaumont 	24c	1012	Lausanne	Waadt	Schweiz	Spitalapotheke, Andere	6011 Verzeichnis a/b/c BetmVV-EDI
Pharmacie Amavita des Deux-Ponts	Galenicare SA	Boulevard Carl-Vogt	4	1205	Genève	Genf	Schweiz	öffentliche Apotheke	6011 Verzeichnis a/b/c BetmVV-EDI
Pharmacie Solidarité	PHARMACIE SOLIDARITE Sàrl	Avenue d'Aïre	38	1203	Genève	Genf	Schweiz	öffentliche Apotheke	6011 Verzeichnis a/b/c BetmVV-EDI
Pharmacies BENU SA		Rue Franche	11	2502	Bienne	Bern	Schweiz	öffentliche Apotheke	6011 Verzeichnis a/b/c BetmVV-EDI

Created small data files for the import.

Field-Names are

• Betriebe: GLN Betrieb Betriebsname 1 Betriebsname 2 Strasse Nummer PLZ Ort Bewilligungskanton Land Betriebstyp BTM Berechtigung
• Personen: GLN Person Name Vorname PLZ Ort Bewilligungskanton Land Diplom BTM Berechtigung Bewilligung Selbstdispensation Bemerkung Selbstdispensation

The XLS files is incomplete as described under http://www.bag.admin.ch/themen/berufe/00411/index.html?lang=de. Using the search mask under http://www.medregom.admin.ch/DE one can find much many more details for a given medical doctor.

After a discussion with Zeno I decided to proceed like this

• harvest all the additional data from medregom using the GLN of a medical doctor (using mechanize/nokogiri). This can be easily tested with a unit test
• create an importer which download the list of all doctors and updates the additional data for each doctor.
• Improve the detail display for a given doctor http://ch.oddb.org/de/gcc/doctor/ean/7601000159199 with the new information from medregom (e.g. add the maplink image). Here it is important to preserve the feedback!. Must be tested with watir.
• harvest all the additional data from medregom using the GLN of hospital/pharmacy
• create an importer which download the list of all hospital/pharmacy and updates the additional data for each institution
• create a new search tab for providers of medical services (Arzt/Spital/Apotheke), listing all hits in alphabetical order ( Must be tested with watirè!)
  • add links for the different types (Arzt/Spital/Apotheke) ( Must be tested with watir!)
• Remove old search tab for Arzt, Spital, Apotheke

Estimated effort to be between 5 and 10 work days.

Verify that import analysis works as expected

After running /usr/local/bin/ruby jobs/update_analysis on thinpower the search via analysis works again on ch.oddb.org. E.g. search as 125Dihydroxycholecalciferol, Alkalische Phosphatase, Alkalische or Phosphatase works, but not as Dihydroxycholecalciferol.

Zeno wants to receive an email if the update completes. Until know we only receive an error if the error fails. Therefore must reorganize the import to catch some useful values (nr updated groups/positions per language).

Pushed the unit test for the analysis import via commit Added unit test for analysis, which was not included in yesterdays commit.

Modifying the unit test to parse a small xlsx file. Reworking the whole unit test to provide tests for two scenarios

• ensure that downloading from bag.admin.ch works
• ensure that a small xlsx file can be parsed correctly

Done with commit Improved unit test for import analysis

Now creating unit test and adating to send a log mail. Works. Introduced a transaction over the whole import to ensure consistency.

Pushed commit Added logging for analysis import. Using transaction.

Mails looks like this

2014-10-14 15:34:39: @archive /var/www/oddb.org/data/xls
2014-10-14 15:34:39: latest_file /var/www/oddb.org/data/xls/analysis_fr_2014.10.14.xlsx is uptodate
2014-10-14 15:34:39: latest_file /var/www/oddb.org/data/xls/analysis_de_2014.10.14.xlsx is uptodate
2014-10-14 15:34:39: deleting all_analysis_group (0 groups)
2014-10-14 15:34:39:  update_group_position de /var/www/oddb.org/data/xls/analysis_de_latest.xlsx
2014-10-14 15:57:34:  update_group_position fr /var/www/oddb.org/data/xls/analysis_fr_latest.xlsx
2014-10-14 16:04:58: update finished with 1118 groups and 1521 positions