Mus fix the the error reported as
Plugin: ODDB::MedwinCompanyPlugin Error: NoMethodError Message: undefined method `keys' for #<Array:0x007f5da2f68540> Backtrace: (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/http/generic_request.rb:34:in `initialize' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/http/request.rb:15:in `initialize' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:1430:in `new' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:1430:in `send_entity' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/http.rb:1218:in `post' (druby://localhost:10006) /var/www/oddb.org/src/util/http.rb:85:in `post' (druby://localhost:10006) /var/www/oddb.org/ext/meddata/src/session.rb:106:in `get_result_list' (druby://localhost:10006) /var/www/oddb.org/ext/meddata/src/drbsession.rb:33:in `search' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1624:in `perform_without_block' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1584:in `perform' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1668:in `block (2 levels) in main_loop' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `loop' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `block in main_loop' (druby://thinpower:37985) /var/www/oddb.org/src/plugin/medwin.rb:72:in `block in update_company' (druby://thinpower:37985) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1624:in `perform_without_block' (druby://thinpower:37985) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1584:in `perform' (druby://thinpower:37985) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1668:in `block (2 levels) in main_loop' (druby://thinpower:37985) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `loop' (druby://thinpower:37985) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `block in main_loop' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/invokemethod.rb:11:in `block_yield' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/invokemethod.rb:18:in `block in perform_with_block' (druby://localhost:10006) /var/www/oddb.org/ext/meddata/src/meddata.rb:15:in `session' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/invokemethod.rb:15:in `perform_with_block' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1582:in `perform' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1668:in `block (2 levels) in main_loop' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `loop' (druby://localhost:10006) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `block in main_loop' /var/www/oddb.org/src/plugin/medwin.rb:71:in `update_company' /var/www/oddb.org/src/plugin/medwin.rb:56:in `block in update' /var/www/oddb.org/src/plugin/medwin.rb:55:in `each_value' /var/www/oddb.org/src/plugin/medwin.rb:55:in `update' /var/www/oddb.org/src/util/updater.rb:599:in `block in update_simple' /var/www/oddb.org/src/util/updater.rb:542:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:597:in `update_simple' /var/www/oddb.org/src/util/updater.rb:408:in `update_medwin_companies' /var/www/oddb.org/src/util/updater.rb:455:in `update_swissmedic_followers' /var/www/oddb.org/src/util/updater.rb:219:in `run' jobs/import_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_daily:12:in `<module:Util>' jobs/import_daily:11:in `<module:ODDB>' jobs/import_daily:10:in `<main>'
The update_company still uses the medwin.ch and not the refdata. Must adapt it. The MEDDATA_URI is used by the plugins medwin, hospitals, vaccines, narcotics, doctors and bsv_xml
Validating via
xmllint --noout --valid oddb_limitation.xml oddb_limitation.xml:2: validity error : Validation failed: no DTD found ! :39:21+0200" VALID_DATE="2017-09-06T14:39:21+0200" GENERATED_BY="oddb2xml 2.4.2"
This does not suprise me as I did see the following lines in my log
U+2265 from UTF-8 to ISO-8859-1: oddb_limitation.xml Ignoring <DSCRD>In Kombination mit Acetylsalicylsäure (ASS) zur Prävention thrombotischer Ereignisse (kardiovaskulärer Tod, Myokardinfarkt, Schlaganfall) bei Patienten mit anamnestisch bekanntem Myokardinfarkt, der mindestens 12 Monate und nicht länger als 2 Jahre zurückliegt, oder bei denen eine allfällige Unterbrechung der dualen Thrombozytenaggregationshemmung nicht länger als 1 Jahr dauerte. Zusätzlich müssen diese Patienten mindestens einen weiteren kardiovaskulären Risikofaktor (zweiter vorangegangener Myokardinfarkt; koronare Mehrgefässerkrankung; Diabetes mellitus, der eine Medikation erfordert; Alter ≥65 Jahre und/oder chronische nichtterminale Niereninsuffizienz) aufweisen.<br>
Therefore I will (for XML files only) always encode to 88591 and back to UTF-8 by replacing bad characters with a '?'.
Zeno suspects Nokogiri. I am writing a small script which reads in a minimal Präparation.xml which contains only the problematic limitation and will look at the encoding before and after passing it to nokogiri.
Using the following script
#!/usr/bin/env ruby
puts RUBY_VERSION
lines = IO.readlines('Preparations.xml')
# content.split("\r\n").each do |line|
lines.each do |line|
begin
iso_8859_1 = line.encode('ISO-8859-1')
rescue => error
puts "#{error}: #{line}"
end
end
I get
2.4.0 U+FEFF from UTF-8 to ISO-8859-1: <?xml version="1.0" encoding="utf-8" standalone="yes"?> U+2265 from UTF-8 to ISO-8859-1: <DescriptionDe>In Kombination mit Acetylsalicylsäure (ASS) zur Prävention thrombotischer Ereignisse (kardiovaskulärer Tod, Myokardinfarkt, Schlaganfall) bei Patienten mit anamnestisch bekanntem Myokardinfarkt, der mindestens 12 Monate und nicht länger als 2 Jahre zurückliegt, oder bei denen eine allfällige Unterbrechung der dualen Thrombozytenaggregationshemmung nicht länger als 1 Jahr dauerte. Zusätzlich müssen diese Patienten mindestens einen weiteren kardiovaskulären Risikofaktor (zweiter vorangegangener Myokardinfarkt; koronare Mehrgefässerkrankung; Diabetes mellitus, der eine Medikation erfordert; Alter ≥65 Jahre und/oder chronische nichtterminale Niereninsuffizienz) aufweisen.<br> U+2019 from UTF-8 to ISO-8859-1: <DescriptionFr>En association avec l'acide acétylsalicylique (AAS), indiqué dans la prévention des événements athéro-thrombotiques (décès d'origine cardio-vasculaire, infarctus du myocarde, accident vasculaire cérébral) chez les patients ayant subi un infarctus du myocarde anamnestique qui remonte à 12 mois au moins et pas plus de 2 ans, ou bien dans l’année suivant l’arrêt d’un traitement précédent par une bithérapie antiplaquettaire. De plus, ces patients doivent présenter au moins un autre facteur de risque cardiovasculaire (deuxième infarctus du myocarde précédent; maladie coronarienne pluritronculaire; diabète sucré qui nécessite une médication; âge ≥ 65 ans et/ou insuffisance rénale chronique non terminale).<br> U+2019 from UTF-8 to ISO-8859-1: <DescriptionIt>In combinazione con l'acido acetilsalicilico (AAS) per la prevenzione degli eventi aterotrombotici (morte cardiovascolare, infarto miocardico e ictus) in pazienti con anamnesi nota di infarto miocardico che risale ad almeno 12 mesi ma che non siano passati più di 2 anni o che una qualsaisi interruzione della terapia duale dell’aggregazione piastrinica non sia durata piu di 1 anno. Inoltre, questi pazienti devono presentare almeno un altro fattore di rischio cardiovascolare (secondo precedente infarto miocardico, malattia coronarica multivascolare, diabete mellito che necessita di trattamento farmacologico, età ≥ 65 anni e/o insufficienza renale cronica non terminale).<br> U+2019 from UTF-8 to ISO-8859-1: Il trattamento ampliato con BRILIQUE nell’arco di 3 anni necessita di un’ulteriore valutazione del profilo rischio-beneficio del paziente.<br>
Therefore we have this problem even before passing the content to Nokogiri. If we can trust https://www.compart.com/de/unicode/U+2265 and https://www.compart.com/de/unicode/charsets/containing/U+2265, this is a valid UTF-8 character, which has no direct equivalent in ISO-8859-1, neither have U+2013 and U+2019. Therefore it is just impossible to translate them to 8859-1.
Therefore I think for XML-File I just do not any conversion at all as we work by default with UTF-8.
For the oddbdat however I must replace the problematic characters U+2013 by '-' and U+2019 by "'".
Fixed this with the following commits:
Running now oddb2xml -e to see, whether the limitation are valid.
After running oddb2xml -e I the oddb_product.xml validates correctly, but oddb_limitation.xml has the following errors
> xsdvalidate oddb2xml.xsd oddb_product.xml oddb_product.xml valid. > xsdvalidate oddb2xml.xsd oddb_limitation.xml oddb_limitation.xml NOT valid. /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:2998:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:3039:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:3079:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:3204:5: error: decimal: Invalid decimal value: expected at least one digit /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:3382:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:4163:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:5197:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:5677:5: error: decimal: Invalid decimal value: expected at least one digit /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:5721:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:6252:5: error: decimal: Invalid decimal value: expected at least one digit /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7061:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7615:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7712:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7740:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7757:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7780:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7803:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7825:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:7847:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:8599:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:9756:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:9855:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:9971:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:10080:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:10908:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:10932:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:10954:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:11705:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:12383:5: error: decimal: Invalid decimal value: expected at least one digit /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:12786:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:12894:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:13478:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration /mnt/oddb-ci2/git/oddb2xml/oddb_limitation.xml:13776:5: error: cvc-complex-type.2.4b: Element not allowed: DEL@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration in element LIM@http://wiki.oddb.org/wiki.php?pagename=Swissmedic.Datendeklaration
We have these DEL elements, beacuse they are no longer valid. It looks that we just never had this case till now. And the oddb2xml.xsd I used to validate is from 2014 whereas the code that emits the DEL is from 2012. Therefore I will leave the XSD as it is and not emit the DEL field, as it was probably never used.
Pushed commits
Released oddb2xml 2.4.3.
The following link does not work as expected http://ch.oddb.org/de/gcc/show/reg/62667.
But these links should be replaced by a correct URL when sending emails. The http is hard-coded in the method resolve_link of src/plugin/swissmedic.rb and src/plugin/plugin.xml. In both cases we use also the constant SERVER_NAME imported from src/config.rb. As there a about 30 occurrences of http://#{SERVER_NAME} I will add a utility method root_url in src/util/session.rb
Changed 18 files. Running jobs/import_swissmedic before pushing commit.