view · edit · sidebar · attach · print · history

Index>

20140526-mail-attachments

Summary

  • replace list of email receipents by configuration data in etc/oddb.yaml
  • estimate effort to create oddb2.csv via oddb2xml

Commits

Index

Keep in Mind

---

Don't export oddbdat in export_daily

Fixed with commit Don't run export oddbdat in export_daily

estimate effort to create oddb2.csv via oddb2xml

Requirements:

  • create oddb2.csv in gem oddb2xml
  • Most (what does this mean) fields must be exported.
  • List fields not exported

The existing oddb.csv contains two different type of records namely #MGrp and #Medi. Here the some examples

Typ;Reg.-Nr.;Packungsnummer;Swissmedic-Nr.;EAN-Code;BSV-Dossier;Pharmacode;Präparat;Galenische Form;Stärke;Packungsgrösse;Numerisch;EFP;PP;Zulassungsinh.;Kat.;SL;Aufnahme in SL;Limitation;Limitationspunkte;Limitationstext;LPPV;Reg.Dat.;Gültig bis;Deaktiviert;Exportprodukt;CAS Reg.Nr.;Generikum;Hat ein Generikum;SB;Ausser Handel (MedRef);Komplementärprodukt;Index Therapeuticus (BAG);Index Therapeuticus (Swissmedic);Betäubungsmittel;Impfstoff/Blutprodukt;In Erneuerung (Swissmedic
#MGrp;A;ALIMENTÄRES SYSTEM UND STOFFWECHSEL
#MGrp;A01;STOMATOLOGIKA
#MGrp;A01A;STOMATOLOGIKA
#Medi;10787;017;10787017;7680107870176;;351372;Sanogencive Plan;Flüssigkeit;233.6 mg/ml;1 à 100 ml;100;;7.90;Laboratoires Plan SA;D;Nein;;;;;Ja;29.07.1941;09.11.2018;;;"";;Nein;;Nein;Ph;13.04.;13.04.;Nein;Nein;Nein

Compare this with line in Packungen xls for Sanogencive Plan 10787 1 Sanogencive Plan, liquide Laboratoires Plan SA 13.04.0. A01A Synthetika human 29.07.1941 29.07.1941 09.11.2018 017 100 ml D myrrha, ratanhiae radix, cochleariae spiritus, camphora racemica myrrha 45.2 mg, ratanhiae radix 15.1 mg, cochleariae spiritus 150.7 mg, camphora racemica 22.6 mg, aromatica, excipiens ad solutionem pro 1 ml, corresp. ethanolum 70-78 % V/V. Pour gargarismes et bains de bouche en cas d'inflammations des gencives

Without digging deeper in the code I don't know how to add support for

  • the MGrp
  • BSV-Dossier
  • Präparat
  • Galenische Form
  • Stärke
  • Packungsgrösse (in Packungen.xls it is Packungsgroesse 100, Einheit ml)
  • EFP
  • PP
  • CAS Reg.Nr.
  • Hat ein Generikum
  • What is the difference between Index Therapeuticus (BAG) and Index Therapeuticus (Swissmedic). And why are they dates?
  • In Erneuerung (Swissmedic

Assuming

  • Impfstoff/Blutprodukt corresponds to the field Heilmittelcode in Packungen.xls, but to give an answer as Ja/Nein we have to interprete the field first.

Support for csv would by adding a format csv (besides dat|xml) via --format=csv.

Alternatively we could emit the dat/xml-files as csv.

Duplicating the method build_dat to build_csv is not very nice, as there are about 15 places with logic about how to generate a field. Therefore it would be probably better to replace the calls of row << content_of_field by something like add_field(content_of_field) with different methods depending on whether we want to generate CSV or DAT files. This would support all fields from oddb.dat.

Adapting oddb2xml to replace the oddb*.xml would be trickier, as we have nested fields, which would need special treatement, as CSV files are flat. This problem is absent when creating substances, limitation, interactions, code, company, person)

Not analysed (fi, fi_product)

Here the lists of nested fields for

  • article
    • ARTCOMP (-> COMPNO)
    • ARTBAR (-> this could be converted easily as it outputs the EAN and 2 constant fields
    • ARTPRI (with 3 sources from swissmedic, zurRose public zurRose price)
    • ARTINS (-> NINCD)
  • product
    • CPT
    • DSCR

Here we would have to solve the problem that for XML we are often skipping element, may a adding else add_empty_field would fix this problem. But we would probably end up with about 10 statement if format == :csv.

replace list of email receipents by configuration data in etc/oddb.yml

Zeno wants first an estimation of effort including a test, whether sending the e-mail works. Recipients to changed would be

  • swissdrug update Swissmedic (xls)
  • oddb.csv
  • oddb2.csv

Things to change would be:

  • add lists for each email destination an entry in etc/oddb.yml, named mail_receivers_for_<recipients_id>
  • Zeno wants it sorted by receipient, which is simpler for his handling, therefore each receipient gets a list of services it is subscriped. This means, that we must add somewhere a helper function which returns a a list of all Receipents for a given services, e.g @@receipents_for_service(service_id)
  • for tests we would add the following line
mail_recipients:
  - ywesee_test@ywesee.com
      - test
      - bsv
      - ipn
      - plugin
      - ouwerkerk
      - swissmedic_journal
      - log # updater
      - sequence # or should we use admin
      - xls
      - dat
      - dat_extended
      - oddb2
      - oddb2csv
      - swissdrug
  • Receipents are define till now in
src/state/admin/sequence.rb:        mail.to = ODDB::State::Admin::Sequence::RECIPIENTS
src/state/admin/sequence.rb:                            smtp.sendmail(mail.encoded, config.smtp_user, [addr] + RECIPIENTS)
src/state/suggest_address.rb:                   RECIPIENTS = [ 'zdavatz@ywesee.com', 'mhatakeyama@ywesee.com' ]
src/state/suggest_address.rb:                                   RECIPIENTS.each { |rec|
src/state/admin/sequence.rb:        mail.to = ODDB::State::Admin::Sequence::RECIPIENT
src/plugin/bsv_xml.rb:    RECIPIENTS = [ 'paul.wiederkehr@pharmasuisse.org' ]
src/plugin/bsv_xml.rb:    BSV_RECIPIENTS = [ 'andrea.frey@bag.admin.ch',
src/plugin/csv_export.rb:    ODDB_RECIPIENTS          = [ "paul.wiederkehr@pharmasuisse.org" ]
src/plugin/csv_export.rb:    ODDB_RECIPIENTS_DAT      = [ "andre.dubied@gmail.com" ]
src/plugin/csv_export.rb:    ODDB_RECIPIENTS_EXTENDED = [ "ouwerkerk@bluewin.ch", "Tim.Suter@just-medical.com" ]
src/plugin/xls_export.rb:               RECIPIENTS = [
src/plugin/swissmedicjournal.rb:                RECIPIENTS = [
src/plugin/ouwerkerk.rb:                RECIPIENTS = [
src/state/suggest_address.rb:                   RECIPIENTS = [ 'zdavatz@ywesee.com', 'mhatakeyama@ywesee.com' ]
src/util/ipn.rb:  RECIPIENTS = [ 'zdavatz@ywesee.com', 'mhatakeyama@ywesee.com', ]
  • Changing the code to use the oddb/configuration should not be difficult nor time consuming (1-4h) as it is already done in a few places. But legacy code always offers surprises, therefore the big spread in estimation
  • Adding a test for sending/receiving emails. I never did it and my experiences is that a lot of stuff can present problems when working with e-mails. Also we have to distinguish the following scenarios
    • oddb-ci: might have other mail system/configuration than thinpower
    • travis-ci: Must we exclude the tests there, add/change configuration
    • thinpower: can we run the test without disrupting other operations
  • Running tests we must be aware, that mail receiving might take some time. I think a setup respecting the following condition might ease the pain considerable
    • unique email-receiver only for testing purposes
    • each test run throws away all pending e-mails at startup
    • logging sending/receipts (via logger and oddb.log) helps debugging (for tests and problems in production)

But adding a test, whether e-mails reach their destination, is the hot spot and should be resolved first.

Therefore total estimation is between 4 hours and 16 hours of effort.

Also we should eliminate all calls to RMail (inside oddb.org) and replace them by calls to Mail.

Starting work. Pushed commit Use Mail instead of (outdated, last release in 2008) RMail for verification of e-mail addresses

Looking at the code I see that we can remove our implementation of Net::SMTP in src/util/smtp_tls.rb. Will replace all occurences of it with direct call to Mail.deliver.

Refactoring to replace the smtp_tls was quite easy, but still took some hours. But quite confident, as the concerned classes all had unit tests (Thanks Hannes!).

Sending on oddb-ci2 does not work as there is no configuration field smtp_auth defined. Must correct this. Was easy. Now resetting the password works. This will be my first watir test to write for tomorrow morning. Afterwards will have to write watir tests for the other occurrences of sending-emails, too.

And we should also fix the warning seen when sending the e-mail

Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.

Hint: use add_charset in Mail::Message when body is given.

view · edit · sidebar · attach · print · history
Page last modified on May 27, 2014, at 08:07 AM