view · edit · sidebar · attach · print · history

< Niklaus.20160322-fix-export-errors | Index | Niklaus.20160316-oddb2xml-prodno >>

20160321-fix-export-error

Summary

  • Fix errors in export, where some packages are nil

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)
  • BSV-Plugin does not send mail. Is it sending two mails at once?
  • Ebixa: bin/admin returns nil for registration('55829').sequences.values.first.iksnr.
  • PI/FI update not correct.

Fix errors in export, where some packages are nil

Must fix errors like

Error: RuntimeError
Message: undefined method `bsv_dossier' for nil:NilClass package ean code=7680580380018
Backtrace:
/var/www/oddb.org/src/view/drugs/csv_result.rb:96:in `rescue in bsv_dossier'
/var/www/oddb.org/src/view/drugs/csv_result.rb:93:in `bsv_dossier'
/var/www/oddb.org/src/view/drugs/csv_result.rb:451:in `block (3 levels) in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:449:in `collect'
/var/www/oddb.org/src/view/drugs/csv_result.rb:449:in `block (2 levels) in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:434:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:434:in `block in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:427:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:427:in `to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:472:in `block in to_csv_file'
/var/www/oddb.org/src/view/drugs/csv_result.rb:471:in `open'
/var/www/oddb.org/src/view/drugs/csv_result.rb:471:in `to_csv_file'
/var/www/oddb.org/src/plugin/csv_export.rb:63:in `_export_drugs'
/var/www/oddb.org/src/plugin/csv_export.rb:30:in `export_drugs'
/var/www/oddb.org/src/util/exporter.rb:102:in `block in export_csv'
/var/www/oddb.org/src/util/exporter.rb:343:in `call'
/var/www/oddb.org/src/util/exporter.rb:343:in `safe_export'
/var/www/oddb.org/src/util/exporter.rb:101:in `export_csv'
/var/www/oddb.org/src/util/exporter.rb:71:in `run'
jobs/export_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/export_daily:12:in `<module:Util>'
jobs/export_daily:11:in `<module:ODDB>'
jobs/export_daily:10:in `<main>'

of

Error: ArgumentError
Message: comparison of NilClass with ODDB::Ean13 failed
Backtrace:
/var/www/oddb.org/src/plugin/csv_export.rb:161:in `sort_by'
/var/www/oddb.org/src/plugin/csv_export.rb:161:in `export_index_therapeuticus'
/var/www/oddb.org/src/util/exporter.rb:168:in `block in export_index_therapeuticus_csv'
/var/www/oddb.org/src/util/exporter.rb:343:in `call'
/var/www/oddb.org/src/util/exporter.rb:343:in `safe_export'
/var/www/oddb.org/src/util/exporter.rb:166:in `export_index_therapeuticus_csv'
/var/www/oddb.org/src/util/exporter.rb:73:in `run'
jobs/export_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/export_daily:12:in `<module:Util>'
jobs/export_daily:11:in `<module:ODDB>'
jobs/export_daily:10:in `<main>'

or

Error: RuntimeError
Message: undefined method `bsv_dossier' for nil:NilClass package ean code=7680580380018
Backtrace:
/var/www/oddb.org/src/view/drugs/csv_result.rb:96:in `rescue in bsv_dossier'
/var/www/oddb.org/src/view/drugs/csv_result.rb:93:in `bsv_dossier'
/var/www/oddb.org/src/view/drugs/csv_result.rb:451:in `block (3 levels) in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:449:in `collect'
/var/www/oddb.org/src/view/drugs/csv_result.rb:449:in `block (2 levels) in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:434:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:434:in `block in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:427:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:427:in `to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:472:in `block in to_csv_file'
/var/www/oddb.org/src/view/drugs/csv_result.rb:471:in `open'
/var/www/oddb.org/src/view/drugs/csv_result.rb:471:in `to_csv_file'
/var/www/oddb.org/src/plugin/csv_export.rb:63:in `_export_drugs'
/var/www/oddb.org/src/plugin/csv_export.rb:42:in `export_drugs_extended'
/var/www/oddb.org/src/util/exporter.rb:105:in `block in export_csv'
/var/www/oddb.org/src/util/exporter.rb:343:in `call'
/var/www/oddb.org/src/util/exporter.rb:343:in `safe_export'
/var/www/oddb.org/src/util/exporter.rb:104:in `export_csv'
/var/www/oddb.org/src/util/exporter.rb:71:in `run'
jobs/export_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/export_daily:12:in `<module:Util>'
jobs/export_daily:11:in `<module:ODDB>'
jobs/export_daily:10:in `<main>'

Created unit tests for the cases where a package can be nil. Fixing the errors. Running jobs/import_bsv_followers to verify that they work. Did not work the first time. Must test for missing sl with unless sl.nil? instead of unless sl

Looking via at the registration 58038 Myozyme, to see why we cannot get the package for GTIN 7680580380018 (package.ikscd is 001). The method introduction_date cannot be executed because the composing slentry is nil. Why? Looks like it is a package recently create as I found in the logs

2016-03-16 22:23:03 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb:969:in `update_package': 58038/01/001 {}
2016-03-16 22:23:03 +0100: /var/www/oddb.org/src/plugin/swissmedic.rb:990:in `update_package': create 58038/01/001 ptr :!create,:!registration,58038!sequence,01!package,001.. package #<ODDB::Package:0x007f0dd6601180

I must check, why the sl_entries did not get updated correctly when running import_bsv.

Fixing via bin/admin

registration('58038').package('001').create_sl_entry
ch.oddb> registration('58038').package('001').sl_entry.valid_from = Date.new(2011,11,1)
-> 2011-11-01
ch.oddb> registration('58038').package('001').sl_entry.introduction_date = Date.new(2011,11,1)
-> 2011-11-01
ch.oddb> registration('58038').package('001').sl_entry.valid_until = Date.new(9999,12,31)
-> 9999-12-31
registration('58038').package('001').odba_store

Now http://ch.oddb.org/de/gcc/show/reg/58038/seq/01/pack/001 displays correctly again. But we cannot edit it as admin user, because the sl_entry.pointer is wrong. Fixing by comparing to another pointer and setting it to the correct value

ch.oddb> registration('53662').sequence('01').package('013').sl_entry.pointer
-> :!registration,53662!sequence,01!package,013!sl_entry.
h.oddb> registration('53662').sequence('01').package('013').sl_entry.pointer == registration('53662').sequence('01').package('013').pointer + :sl_entry
-> true
registration('58038').package('001').sl_entry.pointer = registration('58038').package('001').pointer  + :sl_entry
registration('58038').package('001').odba_store

Now http://ch.oddb.org/de/gcc/sl_entry/reg/58038/seq/01/pack/001 loads correctly again.

As expected by myself this did not fix the problem when exporting oddb.csv, as their are others errors of the same type. Changing the code to insert into the generated CSV values like "missing_sl_entry package ean code=" instead of raising a Runtime Error.

Trying to iterate over all bad sl_entries. Resetting my database to state of yesterday. Adding a breakpoint at line 245 of src/view/admin/package where it displays either "Bearbeiten" or "Erstellen".

Did see the following output of oddbd ODBA::Stub was unable to replace #27558508 from ODDB::Package:#9848279. Model.oid 9848279 at the breakpoint. Model.pointer is :!registration,58038!sequence,01!package,001.. In the pry session we see a difference:

[13] pry(#<ODDB::View::Admin::PackageForm>)> puts 99 unless model.sl_entry
=> nil
[14] pry(#<ODDB::View::Admin::PackageForm>)> puts 99 unless model.sl_entry.nil?
99
=> nil
[16] pry(#<ODDB::View::Admin::PackageForm>)> model.sl_entry.class
=> NilClass

Therefore the admin-interface incorrectly assumes, that there is a SL-Entry even when there is none. Could not continue as pry did throw an exception.

Comparing output of de/gcc/drug/reg/58038/seq/01/pack/001 and /de/gcc/drug/reg/39036/seq/01/pack/001. Here I get

[1] pry(#<ODDB::View::Admin::PackageForm>)> model.oid
=> 34443561
[2] pry(#<ODDB::View::Admin::PackageForm>)> model.pointer
=> :!registration,39036!sequence,01!package,001.
[3] pry(#<ODDB::View::Admin::PackageForm>)> model.sl_entry.class
=> NilClass
[4] pry(#<ODDB::View::Admin::PackageForm>)> model.sl_entry.pointer
NoMethodError: undefined method `pointer' for nil:NilClass
[5] pry(#<ODDB::View::Admin::PackageForm>)> model.sl_entry.nil?
=>
[6] pry(#<ODDB::View::Admin::PackageForm>)> puts model.oid unless model.sl_entry
34443561
=> nil
[7] pry(#<ODDB::View::Admin::PackageForm>)> puts model.oid unless model.sl_entry.nil?
=> nil

I enter a second time into the statement and see

[11] pry(#<ODDB::View::Admin::PackageForm>)> @session.request_path
=> "/de/gcc/drug/reg/58038/seq/01/pack/001"
[12] pry(#<ODDB::View::Admin::PackageForm>)> model.oid
=> 9848279
[13] pry(#<ODDB::View::Admin::PackageForm>)> model.iksnr
=> "58038"
[14] pry(#<ODDB::View::Admin::PackageForm>)> model.class
=> ODDB::Package
[15] pry(#<ODDB::View::Admin::PackageForm>)> model.sl_entry.class
=> NilClass
 "/var/www/oddb.org/src/view/admin/package.rb:246:in `sl_entry'",
[17] pry(#<ODDB::View::Admin::PackageForm>)> continue
ODBA::Stub was unable to replace NilClass#27558508 from ODDB::Package:#9848279

It looks as if we have two stubs (packages) for this. Why? I go therefore into bearbeiten. I get the error about the parent being nil.

I think this problem stems from the fact that when running bsv-xml importer this package got replaced and the sl_entry was not correctly replaced.

Something got really confused. Output of bin/admin

ch.oddb> registration('39036').sequence('01').package('001').sl_entry.nil?
-> true
ch.oddb> registration('58038').sequence('01').package('001').sl_entry.nil?
-> false
ch.oddb> registration('58038').sequence('01').package('001').sl_entry.class
-> NilClass
-> NilClass
ch.oddb> nil.object_id
-> 4
ch.oddb> registration('58038').sequence('01').package('001').sl_entry.object_id
-> 70321544656780
ch.oddb> registration('39036').sequence('01').package('001').sl_entry.object_id
-> 4

Okay. I have an object which pretends to be of class nil, but has a different object_id than nil. Very Ruby!

Finding all of them

ch.oddb> $x = packages.find_all{|pack| pack.sl_entry.kind_of?(NilClass) && pack.sl_entry.object_id != 4}
-> Array
ch.oddb> $x.size
-> 35
$x.each{ |pack| puts "Correcting #{pack.iksnr}/#{pack.seqnr}/#{pack.ikscd}"; pack.sl_entry= nil; pack.odba_store}

Outputs on oddbdd

Correcting 56235/01/014
Correcting 56235/01/016
Correcting 56235/02/018
Correcting 56235/02/020
Correcting 56235/03/022
Correcting 56235/03/024
Correcting 58038/01/001
Correcting 52110/01/130
Correcting 52110/01/149
Correcting 52110/02/041
Correcting 52110/02/076
Correcting 52110/03/084
Correcting 52110/03/106
Correcting 52110/04/157
Correcting 52110/04/165
Correcting 51079/01/028
Correcting 51079/02/029
Correcting 39893/01/053
Correcting 39893/01/061
Correcting 50583/01/023
Correcting 50583/05/112
Correcting 50583/05/120
Correcting 50583/06/147
Correcting 44140/01/012
Correcting 44140/02/020
Correcting 44140/02/039
Correcting 44140/02/047
Correcting 44140/02/225
Correcting 44140/02/249
Correcting 44140/03/071
Correcting 44140/03/101
Correcting 44140/03/233
Correcting 44140/03/241
Correcting 44140/04/209
Correcting 29789/02/045

Okay. It looks as if in the import of the March 16, 35 of the newly created 89 Re-Created missing Packages, the update of the fields from swissmedic_source created the disaster. This item is also a bit special as it is done using a Hash. But now these packages display "Erstellen" for "Sl Eintrag", which is correct, as the bsv_xml-import did not run again.

When looking at http://ch.oddb.org/de/gcc/fachinfo/reg/56235 it display correctly only "Vollst. Fachinformation", "DDD". Rerunning import_bsv_follower another time.

Now seening the following error

Plugin: ODDB::CsvExportPlugin
Error: NoMethodError
Message: undefined method `active?' for #<Array:0x0000000580a1b0>
Backtrace:
/var/www/oddb.org/src/model/package.rb:442:in `public?'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/view/drugs/csv_result.rb:291:in `out_of_trade'
/var/www/oddb.org/src/view/drugs/csv_result.rb:470:in `block (3 levels) in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:468:in `collect'
/var/www/oddb.org/src/view/drugs/csv_result.rb:468:in `block (2 levels) in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:452:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:452:in `block in to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:445:in `each'
/var/www/oddb.org/src/view/drugs/csv_result.rb:445:in `to_csv'
/var/www/oddb.org/src/view/drugs/csv_result.rb:493:in `block in to_csv_file'
/var/www/oddb.org/src/view/drugs/csv_result.rb:492:in `open'
/var/www/oddb.org/src/view/drugs/csv_result.rb:492:in `to_csv_file'
/var/www/oddb.org/src/plugin/csv_export.rb:63:in `_export_drugs'
/var/www/oddb.org/src/plugin/csv_export.rb:30:in `export_drugs'
/var/www/oddb.org/src/util/updater.rb:97:in `block in export_oddb_csv'
/var/www/oddb.org/src/util/updater.rb:530:in `call'
/var/www/oddb.org/src/util/updater.rb:530:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:95:in `export_oddb_csv'
/var/www/oddb.org/src/util/updater.rb:301:in `update_bsv_followers'
jobs/import_bsv_followers:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_bsv_followers:12:in `<module:Util>'
jobs/import_bsv_followers:11:in `<module:ODDB>'
jobs/import_bsv_followers:10:in `<main>'
view · edit · sidebar · attach · print · history
Page last modified on March 22, 2016, at 11:09 AM