view · edit · sidebar · attach · print · history

< Niklaus.20170925-swissmedic-import | Index | Niklaus.20170919-search-limitations >>

20170920-search-limitations

Summary

  • Search selections breaks Price Vergleich
  • Import swissmedic must remove all packages of not listed registrations and update correctly fields like expiration_date
  • Settings do not save correctly default_search
  • Keep in Mind

Commits

Index

Search selections breaks Price Vergleich

Zeno found an example, that the Preis Vergleich still showed outdated packages. The search by Preisvergleich did not list any outdated packages.

But selecting a single product and comparing, e.g. https://ch.oddb.org/de/gcc/compare/ean13/7680660970016 still list https://ch.oddb.org/de/gcc/compare/ean13/7680517950680 (IKNSR 51795 Package 068), which is no longer in Packungen.xlsx.

Verified with bin/admin that the expiration_date/expired? is correctly set

ch.oddb> registration('51795').package('068').expiration_date
-> 2017-05-29
ch.oddb> registration('51795').package('068').expired?
-> true

Therfore I must fix it.

Also in Einstellungen the text for Nur aktuell zugelassene Medikamente is not correctly indented. Fixed with commit Fix identention of 'Nur aktuell zugelassene Medikamente'

To limit the packages shown in the "Preisvergleich" I can use the comparables in src/state/drugs/compare.rb or src/model/package.rb. Which one should I choose.

Resolved the problem with outdate package in Preisvergleich with commit Limit comparable packages to non expired ones

Import swissmedic must remove all packages of not listed registrations and update correctly fields like expiration_date

We remarked yesterday the following two deficiencies:

  1. I shows int the mail 8282 Deactivated Registrations, but there are packages laying around which should not longer exits
  2. The field expiration_date is not correctly updated

The first point was a decision by Hannes Wyss in May 2008 in its commit https://github.com/ngiger/oddb.org/commit/571b16437#diff-f65539e500051180052a1afbec481498R468

In order to minimize the number of rows to compare and to have a nice statistic the swissmedic plugin uses our swissmedic-diff gem.

I think it would not cost so much more time and would be more robust just to compare completely the database entries versus the package.xlsx file. And this should bring the complexity down, as the file is just too large (over 1500 lines). It was less than 500 lines in 2008. And it was less than 900 when I took it over in 2012.

Instead of reporting changes between two XLSX files we would report the changes made to the database, which should ideally be the same. We could for some period continue to show the changes reported by the swissmedic-diff. And/or attach details of the swissmedic-diff reported changes and the changes we applied to the database.

When looking at the file and its git log history, I can resume its features (which will be part of the file after the next commit) like follows

  • update/check can be limited to some IKSRN by passing opts[:iksnrs] (only used by jobs/import_swissmedic_only)
  • create a nice, human readable mail about the changes made
  • ignore drugs only for veterinary use
  • reports missing fachinfos @missing_fis
  • delete inactive_agents
  • delete all packages, but not the sequences when a registration is no longer in the swissmedic packungen (means that the registration was revoked)
  • update all relevant fields, like name, compositions, expiration_date for the corresponding package, sequence and registration
  • uses parslet/compositions from oddb2xml gem to create correct fields for compositions, size, unit, etc
  • atc_code using the best one available (epha, refdata, packungen.xlxs)
  • leaves exactly one copy of the downloaded packungen.xlxs in the format data/xls/Packungen-2017.09.07.xlsx
  • check whether the Gültigkeits-datum (column 'J') is present or not
  • limit its own memory useage to 16 GB

Currently we have the options :check, :fix_galenic_form, :update_compositions, which

  • updated galenic_form only if option fix_galenic_form is given (TODO: ???)
  • optionally check all package (TODO: no longer necessary with new concept)
  • updates existing compositions only if option update_compositions is given (TODO:????)

Must these option still be offered, or should we default to true for all of them if the corresponding fields have changed. Reparsing all compositions takes a long time (around 15) and therefore should probably be avoided.

Starting with creating a human readable data/xls/Packungen-2017.09.20.swisssmedic_diff.

I think that the methods initialize_export_registrations, set_all_export_flag_false, update_export_sequences and update_export_registrations should be grouped together at the end of update and maby even grouped into a separate method to be easily de/activated.

As reading the XLSX files takes minutes, we could use the ssconvert utitily from the Gnumeric application to convert it into a CSV files (takes about 2 seconds) and read in the CSV (which also will take less than a second). Afterwards a diff is easy for CSV files, too.

Alternatively we could also use csvkit from https://csvkit.readthedocs.io/en/1.0.2/index.html. I installed it on oddb-ci2 using sudo pip install csvkit. It took a little bit less than 10 seconds to run in2csv Packungen-2017.09.20.xlsx > Packungen-2017.09.20-1.csv. Comparing the generated csv file I see that csvkit formats date fiels like 2017-08-31 whereas ssconvert produces 2017/08/31.

Zeno has good arguments for the following decision

  • Do not use a not Ruby based tool like gnumeric nor csvkit
  • Keep use of swissmedic-diff gem
  • Settings do not save correctly default_search

This should be fixed with commit Fix saving search_type

Must also fix the error that eg. limiting to SL entries only does not work with drugshortage. Reason: In src/state/drugs/result.rb the package_filter does not get applied.

Found a solution, but it takes a very long time to load the drugshortage list (about 140) seconds. But it can be limited now correctly (including the correct number of packages), eg. to only SL and group A. Must have a next week a closer look at whether my patch Attach:0001-Fix-search-preferences-with-drug_shortage.txt is correct or not.

Removed the unless pack.expired? after line 53 PackageFacade.new(pack, package) in src/state/drugs/compare.rb. Now without "Nur aktuell zugelassene Packungen" ti displays for a trademark show for Similasan 112 entries.After limiting only 82 are shown. Pushed commit Do not limit compare to non expired packages

Drugs with prices, eg.g

ch.oddb> registration('60486').package('002').price_public.class
-> NilClass
ch.oddb> registration('60486').sequence('01').package('002').name_base
-> Omeprazol MUT Sandoz 10 mg
ch.oddb> registration('65990').package('003').name_base
-> Omeprazol Zentiva 10mg
ch.oddb> registration('65990').package('003').price_public.class
-> NilClass
ch.oddb> 
ch.oddb> registration('60493').package('046').price_public.class
-> NilClass
ch.oddb> registration('60493').package('046').name
-> Stadaprazol 10 mg, magensaftresistente Kapseln

The search https://ch.oddb.org/de/gcc/search/zone/drugs/search_query/Omeprazol%20MUT%20Sandoz/search_type/st_sequence?#best_result one finds only prices for the packages 006, 007 and 008, but not for the package 002

view · edit · sidebar · attach · print · history
Page last modified on September 20, 2017, at 08:50 PM