view · edit · sidebar · attach · print · history

20131029-fachinfo-export-yaml

<< | Index | >>


Summary

  • Fix error when running export_fachinfo_yaml

Commits

Index

--

undefined method `barcode' for nil:NilClass while update_swissindex_pharma

Plugin: ODDB::SwissindexPharmaPlugin
Error: NoMethodError
Message: undefined method `barcode' for nil:NilClass
Backtrace:
/var/www/oddb.org/src/plugin/swissindex.rb:189:in `block (2 levels) in
update_out_of_trade'
/var/www/oddb.org/src/plugin/swissindex.rb:189:in `map'
/var/www/oddb.org/src/plugin/swissindex.rb:189:in `block in update_out_of_trade'
/var/www/oddb.org/src/plugin/swissindex.rb:29:in `start'
/var/www/oddb.org/src/plugin/swissindex.rb:185:in `update_out_of_trade'
/var/www/oddb.org/src/plugin/swissindex.rb:164:in `update_package_trade_status'
/var/www/oddb.org/src/util/updater.rb:515:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:491:in `call'
/var/www/oddb.org/src/util/updater.rb:491:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:509:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:426:in
`update_package_trade_status_by_swissindex'
jobs/update_swissindex_pharma:16:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/update_swissindex_pharma:13:in `<module:Util>'
jobs/update_swissindex_pharma:11:in `<module:ODDB>'
jobs/update_swissindex_pharma:10:in `<main>'

We have some really weird stuff in the database, e.g.

ch.oddb> registration('274.0').iksnr
-> 274.0
ch.oddb> registration('274.0').name_base
-> Cardio-Pulmo-Rénal Sérocytol

This can also be seen via http://ch.oddb.org/de/gcc/search/zone/drugs/search_query/Cardio-Pulmo-R%C3%A9nal/search_type/st_sequence#best_result

Pushed commit Conservative programming to avoid errors in update_swissindex_pharma

I think we should call manually in bin/admin delete_registration('274.0') and update the indices afterwards.

composition for sequence 01 not updated from Packungen.xls

continuing work started yesterday

Checking again via bin/admin.

ch.oddb> $suche = []; registrations.each{ |iksnr, reg| $suche << iksnr if reg.sequences['00'] != nil};  $suche.sort!; $suche.size
-> 36
ch.oddb> $suche[0..18]
-> ["00828", "10754", "12498", "17644", "29300", "31919", "33709", "42180", "52265", "57487", "57515", "58458", "62438", "62453", "62470", "62566", "62583", "62612", "62630"]
ch.oddb> $suche[19..36]
-> ["62658", "62672", "62711", "62785", "62786", "62904", "62920", "62976", "63029", "63030", "63200", "63257", "65040", "67769", "69870", "72736", "Zulassungs-Nummer"]

ch.oddb> $suche = []; registrations.each{ |iksnr, reg| $suche << iksnr if reg.sequences['00'] != nil and reg.active?};  $suche.sort!; $suche.size
-> 14
ch.oddb> $suche[0..18]
-> ["62453", "62566", "62612", "62630", "62658", "62672", "62785", "62786", "62904", "62920", "62976", "63029", "63030", "65040"]
ch.oddb> $suche = []; registrations.each{ |iksnr, reg| $suche << iksnr if reg.sequences['00'] != nil and not reg.active?};  $suche.sort!; $suche.size
-> 22
ch.oddb> $suche[0..20]
-> ["00828", "10754", "12498", "17644", "29300", "31919", "33709", "42180", "52265", "57487", "57515", "58458", "62438", "62470", "62583", "62711", "63200", "63257", "67769", "69870", "72736"]
ch.oddb> $suche[20..22]
-> ["72736", "Zulassungs-Nummer"]

ch.oddb> registration('00828').name_base
-> Diencéphale/Zwischenhirn Sérocytol®
ch.oddb> registration('00828').inactive_date
-> 2013-10-24
ch.oddb> registration('00828').active?
-> false
ch.oddb> registration('00828').inactive?
-> true

The IKSNR 00828 (Diencéphale/Zwischenhirn Sérocytol®) has a "Widerruf" for the 24.10.2013. The sequences['00'] must be deleted by running a bin/admin statement like

ch.oddb> $x={}; registrations.each{ |iksnr, reg| if reg.sequences['00'] != nil and not reg.active? then reg.sequences.each{ |id,seq| $x["#{reg.iksnr}.#{id}"]=seq  } end   }; $x.size
-> 22

ch.oddb> registrations.each{ |iksnr, reg| if reg.sequences['00'] != nil and not reg.active? then reg.sequences.each{ |id,seq| delete seq.pointer; update seq.pointer, { :pointer => nil} } end }
-> Hash
ch.oddb> $x={}; registrations.each{ |iksnr, reg| if reg.sequences['00'] != nil and not reg.active? then reg.sequences.each{ |id,seq| $x["#{reg.iksnr}.#{id}"]=seq  } end   }; $x.size
-> 0
ch.oddb> $x=[]; registrations.each{ |iksnr, reg| if reg.sequences['00'] != nil then reg.sequences.each{ |id,seq| $x <<iksnr } end   }; $x.sort!; $x.size
-> 36
ch.oddb> $x[0..18]
-> ["62453", "62453", "62453", "62453", "62453", "62566", "62566", "62612", "62612", "62612", "62612", "62630", "62630", "62630", "62658", "62658", "62672", "62672", "62785"]
ch.oddb> $x[19..36]
-> ["62785", "62785", "62785", "62786", "62786", "62904", "62904", "62904", "62920", "62920", "62976", "63029", "63029", "63030", "63030", "65040", "65040"]

Now we have 36 new entries. This looks okay. Now I have to see, why the update_swissmedic in updater.rb did not correctly handle these situations.

Created a jobs/fix_sequences script. Adding some debug output to see, whether we can use.

Looked at 00556 "Vivotif L, Doppelbeutel" which cannot be found anymore at swissmedinfo.ch but which is in Packungen.xls.

For bisecting the problems I want to know when the duplicate registration (e.g. '556.0') were created.

ch.oddb> registration('556.0').sequences.first[1].package('001').swissmedic_source.first
-> [:import_date, #<Date: 2013-10-14 ((2456580j,0s,0n),+0s,2299161j)>]
ch.oddb> registration('00556').sequences.first[1].package('001').swissmedic_source.first
-> [:import_date, #<Date: 2013-07-06 ((2456480j,0s,0n),+0s,2299161j)>]

Answer is my birthday!

As in the registrations of the form '<y>.0' we don't find a sequence('00') they were probably created when the unchanged algorithm read the new Packungen.xls format. Also Packungen.xls contains exactly 244 lines starting with '00', which is the number of registrations that I found yesterday (see the end of the chapter http://dev.ywesee.com/Niklaus/20131028-swissindex-update#error-export-yaml). This is the double of the 112 $toDelete-fachfinfos. (Each fachinfo produces usually 2 entries in the fachinfo.yaml. One for german and one for french.)

I first test to remove the sequence '00' showed that there quite a few (about 120) registration where we have to delete sequences, e.g

/var/www/oddb.org/src/plugin/swissmedic.rb: 800 2013-10-29 21:34:07 +0100: update_registrations iksnr '00450' seqnr 02 pack 001
/var/www/oddb.org/src/plugin/swissmedic.rb: 830 2013-10-29 21:34:07 +0100: delete packung 001 in seqnr 01

Avoid accessing a nil pointer while running update_swissindex_pharma

Added some debug output to see where the problem stems from.

Made a simple patch to handle the situation where a pointer is nil. See Avoid accessing a nil pointer while running update_swissindex_pharma

Also split the test/util/oddbapp.rb into two files, as minitest seems to have a restriction to 200 test-cases.

running jobs/export_fachinfo_yaml produces undefined method `[]' for nil:NilClass

The problem is that we have a lot (> 900) invalid fachinfos which have no iksnrs attached. After discussing the problem with Zeno we agreed to make a quick patch to just skip exporting fachinfos which have no valid iksnrs.

Fix error when running export_fachinfo_yaml

In my opinion we should delete all fachinfo where the localized name is empty and no iksnrs are defined. Running in bin/admin

ch.oddb> $toDelete = []; fachinfos.each{ |iksnr, fi| $toDelete << fi.pointer if fi.iksnrs.size == 0 and fi.localized_name == nil}; $toDelete.size
> 112
ch.oddb> $toDelete = []; fachinfos.each{ |iksnr, fi| delete fi.pointer if fi.iksnrs.size == 0 and fi.localized_name == nil}; $toDelete.size
> 112

Now an export of the fachinfo completes without any error.

I'm wondering whether we have problems with other FI having no iksnrs. Printing them via fachinfos.each{ |iksnr, fi| $stdout.puts fi.localized_name if fi.iksnrs.size == 0 }; gives Attach:fi_with_no_iksnrs.txt with 978 entries.

Looking at two examples are fachinfos.each{ |iksnr, fi| puts "xx #{fi.localized_name} ik #{fi.iksnrs.inspect}" if fi.localized_name.eql?('Levocetirizin-Mepha®') }; returns 5 fis, 4 with no iksrns, one with 62519 fachinfos.each{ |iksnr, fi| puts "xx #{fi.localized_name} ik #{fi.iksnrs.inspect}" if fi.localized_name.eql?('Famvir®') }; returns 4 fis, all with no iksrns. The IKSNR should be 52693. I am running an import for 52693 and 62519. Situations is unchanged. Now I delete the pointers with fachinfos.each{ |iksnr, fi| delete fi.pointer if fi.localized_name.eql?('Famvir®') }; and run the import script again.

I am concerned that fachinfos are not correctly updated after this import.

ch.oddb> $y=[]; fachinfos.each{ |iksnr, fi| $y << "a1  #{fi.localized_name} ik #{fi.iksnrs.inspect}" if fi.localized_name.index('Levocetirizin') }; $y
-> ["a1  Levocetirizin Sandoz® ik []", "a1  Levocetirizin Sandoz® ik []", "a1  Levocetirizin Spirig HC® ik [\"62920\"]", "a1  Levocetirizin Sandoz® ik [\"62522\"]"]
ch.oddb> fi = registration('62519').fachinfo; $y = "a3 #{fi.localized_name} ik #{fi.iksnrs.inspect}"  if fi.localized_name.index('Levocetirizin'); $y
-> a3 Levocetirizin-Mepha® ik ["62519"]

The namebase of a fachinfo can be nil if this no longer active. But in this case iksnrs must still contain at least one entry.

I think we should delete all fachinfos where the name_base == nil and iksnrs.size == 0. But first we must verify that we have not active registration where the fachinfo.name_base is nil.

ch.oddb> $y=[];registrations.each{ |iksnr, reg| $y << reg if reg.fachinfo and reg.active? and reg.fachinfo.name_base == nil}; $y.size
-> 0
ch.oddb> $y=[];registrations.each{ |iksnr, reg| $y << reg if reg.fachinfo and reg.fachinfo.name_base == nil}; $y.size
-> 32
ch.oddb> $y[0].iksnr
-> 42009
ch.oddb> $y[0].fachinfo.iksnrs
-> ["42009"]
ch.oddb> $y=[];registrations.each{ |iksnr, reg| $y << reg if reg.fachinfo and reg.fachinfo.iksnrs.size == 0}; $y.size
-> 0

I think we should delete all fachinfos where the name_base == nil. But first we must verify that we have the fachinfo.oid of all registration can be found via fachinfo(oid).

ch.oddb> $y=[];registrations.each{ |iksnr, reg| $y << reg if reg.fachinfo and fachinfo(reg.fachinfo.oid) == nil}; $y.size
-> 0
ch.oddb> $y=[];registrations.each{ |iksnr, reg| $y << reg if reg.fachinfo and reg.fachinfo.iksnrs.size == 0}; $y.size
-> 0
ch.oddb> $toDelete=[];fachinfos.each{ |iksnr, fi| $toDelete << fi if fi.iksnrs.size == 0}; $toDelete.size
-> 966
ch.oddb> fachinfos.each{ |iksnr, fi| delete fi.pointer if fi.iksnrs.size == 0};

Deleting takes quite a while. Now I would be very grateful if I had a test suite which would allow me to verify that I did not break something vital!

These fachinfo are not part of the orphanded_fachinfos, as demonstrated by this script

ch.oddb> $all=[]; orphaned_fachinfos.each{ |oid, fi| $all << oid if fachinfo(oid) }; $all.size
-> 0
view · edit · sidebar · attach · print · history
Page last modified on October 30, 2013, at 08:21 AM