view · edit · sidebar · attach · print · history

Index>

20160113-fix-migel-import

Summary

  • Migel imports don't work anymore
  • Why is Fachinfo for 65690 Kyprolis not updated in import_daily?
  • Exclude ATC-codes from inactive products, e.g. Adenosin

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)

Why is Fachinfo for 65690 Kyprolis not updated in import_daily?

Zeno remarked, that today no new FI were imported, even when Kyprolis was added. Why?

When searching for 65690 in log/oddb/debug/2016/01.log I found 

2016-01-12 11:18:01 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb:335 delete object #<ODDB::Package:0x00000110974a30 @swissmedic_source={:import_date=>#<Date: 2014-12-04 ((2456996j,0s,0n),+0s,2299161j)>, :iksnr=>"62905", :seqnr=>"01", :name_base=>"Drospibel 30, compresse rivestite con film", :company=>"EFFIK SA", :index_therapeuticus=>"09.02.1.", :atc_class=>"G03AA12", :production_science=>"Synthetika human", :registration_date=>#<DateTime: 2014-06-04T00:00:00+00:00 ((2456813j,0s,0n),+0s,2299161j)>, :sequence_date=>#<DateTime: 2014-06-04T00:00:00+00:00 ((2456813j,0s,0n),+0s,2299161j)>, :expiry_date=>#<DateTime: 2019-06-03T00:00:00+00:00 ((2458638j,0s,0n),+0s,2299161j)>, :ikscd=>"001", :size=>"1 x 21", :unit=>"Tablette(n)", :ikscat=>"B", :substances=>"drospirenonum, ethinylestradiolum", :composition=>"drospirenonum 3 mg, ethinylestradiolum 0.03 mg, excipiens pro compresso obducto.", :indication_registration=>"Contraccettivo ormonale", :indication_sequence=>nil}, @oid=31965715, @data_origins={"ikscat"=>:swissmedic, "swissmedic_source"=>:swissmedic, "refdata_override"=>:refdata, "out_of_trade"=>:refdata, "pharmacode"=>:bag}, @odba_persistent=true, @ikscat="B", @revision=2014-12-04 22:21:36 +0100, @sequence=#<ODBA::Stub:2286657020#31965707 @odba_class= @odba_container=2286658840#31965715>, @odba_id=31965715, @parts=#<ODBA::Stub:2286656900#31965716 @odba_class= @odba_container=2286658840#31965715>, @odba_observers=[], @pointer=:!registration,62905!sequence,01!package,001., @ikscd="001", @feedbacks=#<ODBA::Stub:2286690260#31965717 @odba_class= @odba_container=2286658840#31965715>, @refdata_override=false, @out_of_trade=false, @pharmacode="6067005", @mail_order_prices=#<ODBA::Stub:2286690140#31965718 @odba_class= @odba_container=2286658840#31965715>>
2016-01-12 11:27:28 +0100: deactivate_not_found_package check "65690". 34 @deletes_packages
2016-01-13 07:03:39 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1172:in `block in parse_and_update': parse_and_update: must add fachinfo for ["65690"]
2016-01-13 07:03:39 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1188:in `block in parse_and_update': parse_and_update: calls parse_fachinfo reparse dist /var/www/oddb.org/data/html/fachinfo/de/Kyprolis__swissmedicinfo.html false iksnrs_from_xml ["65690"] Kyprolis__swissmedicinfo.html, name Kyprolis® de title Kyprolis®
2016-01-13 07:03:39 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1198:in `parse_and_update': fachinfo empty? content true false iksnrs_from_xml ["65690"] dist /var/www/oddb.org/data/html/fachinfo/de/Kyprolis__swissmedicinfo.html
2016-01-13 07:03:39 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1207:in `parse_and_update': calling update_fachinfo Kyprolis®, ["65690"]
2016-01-13 07:03:39 +0100: /var/www/oddb.org/src/plugin/text_info.rb:248:in `update_fachinfo': update_fachinfo Kyprolis® iksnr ["65690"]
2016-01-13 07:03:39 +0100: /var/www/oddb.org/src/plugin/text_info.rb:267:in `block in update_fachinfo': update_fachinfo Kyprolis® iksnr 65690 store_fachinfo {} [:de] ATC L01XX45
2016-01-13 07:04:46 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1172:in `block in parse_and_update': parse_and_update: must add fachinfo for ["65690"]
2016-01-13 07:04:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1188:in `block in parse_and_update': parse_and_update: calls parse_fachinfo reparse dist /var/www/oddb.org/data/html/fachinfo/fr/Kyprolis__swissmedicinfo.html false iksnrs_from_xml ["65690"] Kyprolis__swissmedicinfo.html, name Kyprolis® fr title Kyprolis®
2016-01-13 07:04:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1198:in `parse_and_update': fachinfo empty? content true false iksnrs_from_xml ["65690"] dist /var/www/oddb.org/data/html/fachinfo/fr/Kyprolis__swissmedicinfo.html
2016-01-13 07:04:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1207:in `parse_and_update': calling update_fachinfo Kyprolis®, ["65690"]
2016-01-13 07:04:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:248:in `update_fachinfo': update_fachinfo Kyprolis® iksnr ["65690"]
2016-01-13 07:04:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:267:in `block in update_fachinfo': update_fachinfo Kyprolis® iksnr 65690 store_fachinfo {} [:fr] ATC

With a close look I found 

2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1172:in `block in parse_and_update': parse_and_update: must add fachinfo for ["65690"]                                                
2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1188:in `block in parse_and_update': parse_and_update: calls parse_fachinfo reparse dist /var/www/oddb.org/data/html/fachinfo/de/Kypro
2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1198:in `parse_and_update': fachinfo empty? content true false iksnrs_from_xml ["65690"] dist /var/www/oddb.org/data/html/fachinfo/de/
2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1202:in `block in parse_and_update': _infos de #<ODDB::FachinfoDocument2001:0x00000055b0b7d8> ODDB::FachinfoDocument2001              
2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1202:in `block in parse_and_update': _infos fr  NilClass                                                                              
2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1207:in `parse_and_update': calling update_fachinfo Kyprolis®, ["65690"]                                                              
2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:248:in `update_fachinfo': update_fachinfo Kyprolis® iksnr ["65690"]                                                                   
2016-01-13 02:03:47 +0100: /var/www/oddb.org/src/plugin/text_info.rb:267:in `block in update_fachinfo': update_fachinfo Kyprolis® iksnr 65690 store_fachinfo {} [:de] ATC L01XX45     

2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1172:in `block in parse_and_update': parse_and_update: must add fachinfo for ["65690"]                                                
2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1188:in `block in parse_and_update': parse_and_update: calls parse_fachinfo reparse dist /var/www/oddb.org/data/html/fachinfo/fr/Kypro
2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1198:in `parse_and_update': fachinfo empty? content true false iksnrs_from_xml ["65690"] dist /var/www/oddb.org/data/html/fachinfo/fr/
2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1202:in `block in parse_and_update': _infos de  NilClass                                                                              
2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1202:in `block in parse_and_update': _infos fr #<ODDB::FachinfoDocument2001:0x00000057ebe108> ODDB::FachinfoDocument2001              
2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:1207:in `parse_and_update': calling update_fachinfo Kyprolis®, ["65690"]                                                              
2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:248:in `update_fachinfo': update_fachinfo Kyprolis® iksnr ["65690"]                                                                   
2016-01-13 02:03:56 +0100: /var/www/oddb.org/src/plugin/text_info.rb:267:in `block in update_fachinfo': update_fachinfo Kyprolis® iksnr 65690 store_fachinfo {} [:fr] ATC

What is going on here. Why it the FI not stored correctly? Looking in bin/admin show that the fachfino is nil 

ch.oddb> registration('65690').fachinfo.class
-> NilClass

Looks like my patch to avoid errors when IKSNR is 0 was wrong! Yes, very stupid error. unless iksnr.to_i != 0 must be corrected to unless iksnr.to_i == 0. Rerunning import_daily locally to verify.

Pushed commit Corrected comparision to 0

Migel imports don't work anymore

The migel import takes a very long time (510) seconds. I think somewhere the algorithm is wasting a lot of time. But with my changes of yesterday I did see, that http://oddb-ci2.dyndns.org/fr/gcc/migel_search/migel_product/010101001 contained the new item 3666142 BIBI Einhandbrustpumpe when comparing it to the state at ch.oddb.org. But the generated files migel_products_<lang>.csv where empty. Must investigate why my seperating broke the work. Will continue to work on doing the download before each save_all.

Do I get the right data? As I changed last year downloading from https://index.ws.e-mediat.net/Swissindex/NonPharma/ws_NonPharma_V101.asmx?WSDL to http://refdatabase.refdata.ch/Service/Article.asmx?WSDL I get article/items in file (/var/www/oddb.org/data/xml/XMLRefdataNonPharma-2016.01.13.xml) like 

      <ITEM DT="2016-01-13T00:00:00">
        <ATYPE>NONPHARMA</ATYPE>
        <GTIN>7612479001117</GTIN>
        <PHAR>1624501</PHAR>
        <NAME_DE>AMEDA Einhandmilchpumpe mit Flexishield</NAME_DE>
        <NAME_FR>AMEDA tire lait une main avec Flexishield</NAME_FR>
        <AUTH_HOLDER_NAME>Nufer Medical AG</AUTH_HOLDER_NAME>
        <AUTH_HOLDER_GLN>2000000086125</AUTH_HOLDER_GLN>
      </ITEM>

Here I don't have an italian name. The data from e-mediat-net (/var/www/migel/data/xml/XMLSwissindexNonPharma-DE-2015.02.02.xml) had the following format 

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <soap:Body>
    <NONPHARMA xmlns="http://swissindex.e-mediat.net/SwissindexNonPharma_out_V101" CREATION_DATETIME="2015-02-02T18:10:15.0280259+01:00">
<..>
      <ITEM DT="2014-10-17T00:00:00">
        <GTIN>7612479001117</GTIN>
        <PHAR>1624501</PHAR>
        <STATUS>A</STATUS>
        <STDATE>2010-07-09T00:00:00</STDATE>
        <LANG>DE</LANG>
        <DSCR>AMEDA Einhandmilchpumpe mit Flexishield</DSCR>
        <ADDSCR>1 Stk</ADDSCR>
        <COMP>
          <NAME>Nufer Medical AG </NAME>
        </COMP>
      </ITEM>
<..>

But where does the 3666142 BIBI Einhandbrustpumpe come from? I don't have it in XMLRefdataNonPharma-2016.01.13.xml. Nor did I find 3666142 in any of the oddb2xml files.

Answer: Query as done via in class SwissindexMigel method search_migel in from oddb.org/src/ext/swissindex/src/swissindex.rb.

Exclude ATC-codes from inactive products, e.g. Adenosin

Currently a search for Adenosin returns not only products with ATC-Code C01EB10, but also V07AB (Lösungs- und Verdünnungsmittel, inkl. Spüllösungen), because the IKSNR 57387 (no longer active) referenced it.

Currently in oddb_app.rb we have 

  def search_exact_substance(query)
    sequences = ODBA.cache.retrieve_from_index('substance_index_sequence', query)
    _search_exact_classified_result(sequences, :substance)
  end

Collecting only the active substances did not work. bin/admin still reports 3 ATC-classes for search_combined('Adenosin', 'de').atc_classes, search by price (aka search_exact_substance) is now correct. 

ch.oddb> search_exact_substance('Adenosin').atc_classes                                                             
-> [Adenosin]
ch.oddb> search_combined('Adenosin').atc_classes       
-> wrong number of arguments (1 for 2)
ch.oddb> search_combined('Adenosin', 'de').atc_classes
-> [Andere Diagnostika, Adenosin, Lösungs- und Verdünnungsmittel, inkl. Spüllösungen]

Or does search_by_atc(key) return too many Trying to fix substance_index_sequence. Calling jobs/rebuild_indices substance_index_sequence. But the problems lays in search_by_substance which uses the substance_index_atc which returns also the ATC of no longer active sequences.

Adding a new methode active_substances in util/oddbapp.rb and using it for indices substance_index_sequence sequence_index_atc. Rebuilding indices. Did not work. Must correct index substance_index_atc, too.

Trying to replace definition of sequence_index_atc with 

-resolve_target: 'sequences.collect { |seq| seq.atc_class }.uniq'
+resolve_target: 'sequences.collect { |seq| seq.atc_class if seq.active?}.compact.uniq'

expect that bin/admin returns only 1 element when calling search_by_substance('Adenosin'). Currently it returns 3 elements. While rebuilding the index it returns an empty array. Still returned 3 

ch.oddb> search_by_substance('Adenosin')
-> [Andere Diagnostika, Adenosin, Lösungs- und Verdünnungsmittel, inkl. Spüllösungen]
ch.oddb> search_by_substance('Adenosin').size
-> 3

Patching also resolve_origin 

 resolve_search_term: 'search_keys'
-resolve_target: 'sequences.collect { |seq| seq.atc_class }.uniq'
-resolve_origin: 'sequences.collect { |seq| seq.active_agents.collect { |agent| agent.substance } }.uniq'
+resolve_target: 'sequences.collect { |seq| seq.atc_class if seq.active?}.compact.uniq'
+resolve_origin: 'sequences.collect { |seq| seq.active_agents.collect { |agent| agent.substance }  if seq.active? }.uniq'

Calling jobs/rebuild_indices substance_index_atc. Even before finishing, it looks that it does the job. Waiting for rebuild_indices to finish. Will then drop patching the resolve_target patch to see, whether is really needed.

Another example is Gentamycin which is found via 

ch.oddb> search_by_interaction('Gentamycin', 'de').atc_classes
-> [Cytarabin, Kombinationen von Antibiotika, Aztreonam, Furosemid, Linezolid, Ketorolac, Ciclosporin, Vancomycin, Alprostadil, Dexamethason und Antiinfektiva]

The following results are expected. Iscover should return nothing, Adenosin should return one ATC-Code. Gentamycin should return less results. Here the problem lays that the price search also looks in the interaction where Gentamycin is still mentioned, even if it is no longer active. Therefore I could see only a difference using bin/admin. Actually it return 

ch.oddb> search_by_interaction('Gentamycin','de').atc_classes
-> [Furosemid, Ciclosporin, Linezolid, Aztreonam, Ketorolac, Vancomycin, Alprostadil, Dexamethason und Antiinfektiva, Kombinationen von Antibiotika, Cytarabin]
ch.oddb> search_by_interaction('Gentamycin','de').atc_classes.size
-> 10

with my change it return 

ch.oddb> search_by_interaction('Gentamycin','de').atc_classes
> [Cytarabin, Kombinationen von Antibiotika, Aztreonam, Furosemid, Linezolid, Ketorolac, Ciclosporin, Vancomycin, Alprostadil]
ch.oddb> search_by_interaction('Gentamycin','de').atc_classes.size
-> 9

Therefore I leave my changes for the interaction_index_de|fr. Added test-cases in spec/searchbar_spec.rb. Pushed commit Consider only active sequences for searches

view · edit · sidebar · attach · print · history
Page last modified on January 15, 2016, at 12:00 PM