view · edit · sidebar · attach · print · history

< Niklaus.20160113-fix-migel-import | Index | Niklaus.20160111-missing-cool-in-vaccines >>

20160112-fix-migel-import

Summary

  • Migel imports don't work anymore
  • Fix NotImplementedError in rss update
  • Remove unsed 'E'/12 from oddb_code.xml

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)

Migel imports don't work anymore

One of my commits broke running the Migel imports. E.g

NoMethodError
undefined method `session' for ODDB::Swissindex:Module
(druby://localhost:50002) /usr/local/lib/ruby/1.9.1/drb/invokemethod.rb:14:in `perform_with_block'
(druby://localhost:50002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1506:in `perform'
(druby://localhost:50002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:50002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:50002) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/migel/lib/migel/plugin/swissindex.rb:36:in `block in save_all_products'
/usr/local/lib/ruby/1.9.1/csv.rb:1367:in `open'
/var/www/migel/lib/migel/plugin/swissindex.rb:35:in `save_all_products'
/var/www/migel/lib/migel/util/importer.rb:350:in `save_all_products'
/var/www/migel/lib/migel/util/importer.rb:234:in `reported_save_all_products'
/var/www/migel/lib/migel/util/importer.rb:224:in `block in save_all_products_all_languages'
/var/www/migel/lib/migel/util/importer.rb:221:in `each'
/var/www/migel/lib/migel/util/importer.rb:221:in `save_all_products_all_languages'
jobs/update_migel_products_with_report:13:in `block in <module:Util>'
/var/www/migel/lib/migel/util/job.rb:22:in `call'
/var/www/migel/lib/migel/util/job.rb:22:in `run'
jobs/update_migel_products_with_report:12:in `<module:Util>'
jobs/update_migel_products_with_report:11:in `<module:Migel>'
jobs/update_migel_products_with_report:10:in `<main>'

The migel import do not work anymore, because some classes were renamed and not tested. When removing ext/swissindex/bin/swissindex_(non)pharmad, I did not think of the migel import and update_migel_products_with_report is run only on 1 of January and June. The commit Changed URL for refdata changed the names SwissindexPharma and SwissindexNonpharma -> SwissindexMigel. But no corresponding change in the migel repository was done.

Made some changes and the update created several files, but crashed with

2016-01-12 10:23:04 +0100: update_group ["99", "50", "01", "00", "1"] groupcd 99 de
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:525: [BUG] Segmentation fault
ruby 1.9.3p545 (2014-02-24 revision 45159) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0028 p:0020 s:0101 b:0099 l:000086 d:000098 BLOCK  /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:525
c:0027 p:---- s:0096 b:0096 l:000095 d:000095 FINISH
c:0026 p:---- s:0094 b:0094 l:000089 d:000093 IFUNC 
c:0025 p:---- s:0092 b:0092 l:000091 d:000091 CFUNC  :each
c:0024 p:---- s:0090 b:0090 l:000089 d:000089 CFUNC  :any?
c:0023 p:0040 s:0087 b:0087 l:000086 d:000086 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:524
c:0022 p:0051 s:0083 b:0083 l:000082 d:000082 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:37
c:0021 p:0024 s:0076 b:0076 l:000075 d:000075 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:72
c:0020 p:0028 s:0073 b:0073 l:000072 d:000072 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:309
c:0019 p:0018 s:0066 b:0066 l:0022b8 d:0022b8 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:225
c:0018 p:0056 s:0061 b:0061 l:000060 d:000060 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:45
c:0017 p:0014 s:0057 b:0057 l:000046 d:000056 BLOCK  /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:97
c:0016 p:---- s:0052 b:0052 l:000051 d:000051 FINISH
c:0015 p:---- s:0050 b:0050 l:000049 d:000049 CFUNC  :delete_if
c:0014 p:0073 s:0047 b:0047 l:000046 d:000046 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:96
c:0013 p:0049 s:0042 b:0042 l:000028 d:000041 BLOCK  /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:105
c:0012 p:---- s:0039 b:0039 l:000038 d:000038 FINISH
c:0011 p:---- s:0037 b:0037 l:000036 d:000036 CFUNC  :each_value
c:0010 p:0011 s:0034 b:0034 l:000028 d:000033 BLOCK  /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:102
c:0009 p:0019 s:0032 b:0032 l:000031 d:000031 METHOD <internal:prelude>:10
c:0008 p:0050 s:0029 b:0029 l:000028 d:000028 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:101
c:0007 p:0110 s:0021 b:0021 l:000020 d:000020 METHOD /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:79
c:0006 p:0030 s:0014 b:0014 l:000968 d:000013 BLOCK  /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:490
c:0005 p:---- s:0011 b:0011 l:000010 d:000010 FINISH
c:0004 p:---- s:0009 b:0009 l:000008 d:000008 CFUNC  :loop
c:0003 p:0040 s:0006 b:0006 l:000968 d:000005 BLOCK  /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:487
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH
c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP   

-- Ruby level backtrace information ----------------------------------------
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:487:in `block in start_cleaner'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:487:in `loop'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:490:in `block (2 levels) in start_cleaner'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:79:in `clean'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:101:in `_clean'
<internal:prelude>:10:in `synchronize'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:102:in `block in _clean'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:102:in `each_value'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:105:in `block (2 levels) in _clean'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:96:in `odba_retire'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:96:in `delete_if'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:97:in `block in odba_retire'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:45:in `odba_id2ref'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:225:in `fetch'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache.rb:309:in `fetch_or_do'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:72:in `_odba_object'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/cache_entry.rb:37:in `object_id2ref'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:524:in `odba_unsaved?'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:524:in `any?'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:524:in `each'
/usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/persistable.rb:525:in `block in odba_unsaved?'

-- C level backtrace information -------------------------------------------
/usr/lib64/libruby19.so.1.9(+0x159cec) [0x7f32f374ccec]
/usr/lib64/libruby19.so.1.9(+0x5a119) [0x7f32f364d119]
/usr/lib64/libruby19.so.1.9(rb_bug+0xb7) [0x7f32f364da47]
/usr/lib64/libruby19.so.1.9(+0xf939e) [0x7f32f36ec39e]
/lib64/libc.so.6(+0x38500) [0x7f32f3281500]
/usr/lib64/libruby19.so.1.9(+0x1444f8) [0x7f32f37374f8]
/usr/lib64/libruby19.so.1.9(+0x146721) [0x7f32f3739721]
/usr/lib64/libruby19.so.1.9(+0x14e9e7) [0x7f32f37419e7]
/usr/lib64/libruby19.so.1.9(+0x14f3f3) [0x7f32f37423f3]
/usr/lib64/libruby19.so.1.9(+0x5436e) [0x7f32f364736e]
/usr/lib64/libruby19.so.1.9(+0x147856) [0x7f32f373a856]
/usr/lib64/libruby19.so.1.9(rb_yield+0x4db) [0x7f32f374861b]
/usr/lib64/libruby19.so.1.9(rb_ary_each+0x52) [0x7f32f3620862]
/usr/lib64/libruby19.so.1.9(+0x1504e7) [0x7f32f37434e7]
/usr/lib64/libruby19.so.1.9(rb_iterate+0xd5) [0x7f32f3739da5]
/usr/lib64/libruby19.so.1.9(rb_block_call+0x2d) [0x7f32f3739f6d]
/usr/lib64/libruby19.so.1.9(+0x53970) [0x7f32f3646970]
/usr/lib64/libruby19.so.1.9(+0x152dc1) [0x7f32f3745dc1]
/usr/lib64/libruby19.so.1.9(+0x149272) [0x7f32f373c272]
/usr/lib64/libruby19.so.1.9(+0x14f3f3) [0x7f32f37423f3]
/usr/lib64/libruby19.so.1.9(rb_yield_values+0x2fe) [0x7f32f374895e]
/usr/lib64/libruby19.so.1.9(+0x734dc) [0x7f32f36664dc]
/usr/lib64/libruby19.so.1.9(+0x72782) [0x7f32f3665782]
/usr/lib64/libruby19.so.1.9(st_foreach+0xd1) [0x7f32f36f4871]
/usr/lib64/libruby19.so.1.9(+0x73a0a) [0x7f32f3666a0a]
/usr/lib64/libruby19.so.1.9(rb_ensure+0xbb) [0x7f32f3651d7b]
/usr/lib64/libruby19.so.1.9(rb_hash_foreach+0x36) [0x7f32f3666fa6]
/usr/lib64/libruby19.so.1.9(rb_hash_delete_if+0x27) [0x7f32f36680b7]
/usr/lib64/libruby19.so.1.9(+0x152dc1) [0x7f32f3745dc1]
/usr/lib64/libruby19.so.1.9(+0x149272) [0x7f32f373c272]
/usr/lib64/libruby19.so.1.9(+0x14f3f3) [0x7f32f37423f3]
/usr/lib64/libruby19.so.1.9(rb_yield+0x26e) [0x7f32f37483ae]
/usr/lib64/libruby19.so.1.9(+0x72442) [0x7f32f3665442]
/usr/lib64/libruby19.so.1.9(+0x72782) [0x7f32f3665782]
/usr/lib64/libruby19.so.1.9(st_foreach+0xd1) [0x7f32f36f4871]
/usr/lib64/libruby19.so.1.9(+0x73a0a) [0x7f32f3666a0a]
/usr/lib64/libruby19.so.1.9(rb_ensure+0xbb) [0x7f32f3651d7b]
/usr/lib64/libruby19.so.1.9(rb_hash_foreach+0x36) [0x7f32f3666fa6]
/usr/lib64/libruby19.so.1.9(+0x741be) [0x7f32f36671be]
/usr/lib64/libruby19.so.1.9(+0x152dc1) [0x7f32f3745dc1]
/usr/lib64/libruby19.so.1.9(+0x149272) [0x7f32f373c272]
/usr/lib64/libruby19.so.1.9(+0x14f3f3) [0x7f32f37423f3]
/usr/lib64/libruby19.so.1.9(+0x1543d3) [0x7f32f37473d3]
/usr/lib64/libruby19.so.1.9(rb_rescue2+0x16b) [0x7f32f3651a0b]
/usr/lib64/libruby19.so.1.9(+0x145c7e) [0x7f32f3738c7e]
/usr/lib64/libruby19.so.1.9(+0x152dc1) [0x7f32f3745dc1]
/usr/lib64/libruby19.so.1.9(+0x149272) [0x7f32f373c272]
/usr/lib64/libruby19.so.1.9(+0x14f3f3) [0x7f32f37423f3]
/usr/lib64/libruby19.so.1.9(+0x150152) [0x7f32f3743152]
/usr/lib64/libruby19.so.1.9(+0x1606ee) [0x7f32f37536ee]
/usr/lib64/libruby19.so.1.9(+0x1607d7) [0x7f32f37537d7]
/lib64/libpthread.so.0(+0x91a7) [0x7f32f30341a7]
/lib64/libc.so.6(clone+0x6d) [0x7f32f333720d]

-- Other runtime information -----------------------------------------------

<...>

The produced files migel_products_<lang>.csv contain only the header. MiGeL-2016.01.12.xls looks okay and contains the changes mentioned by the BAG in http://www.bag.admin.ch/themen/krankenversicherung/00263/00264/04184/index.html?lang=de&download=NHzLpZeg7t,lnp6I0NTU042l2Z6ln1acy4Zn4Z2qZpnO2Yuq2Z6gpJCMe4B4g2ym162epYbg2c_JjKbNoKSn6A--

One of the changes is that position 21.05.02.03.2 was superseded 21.05.02.03.3. Looking whether I can find the new item in the database.

Verifying with bin/admin of oddb.org

ch.oddb> search_migel_products('21.05.02.03.3','de')
-> [Miete/Monat]
ch.oddb> search_migel_products('21.05.02.03.2','de')
-> [Miete/Monat]
ch.oddb> search_migel_products('21.05.02.03.4','de')
-> []                                                                                                                                                                                                      

Running jobs/update_migel_products_with_report a second time finished without sending any e-mail and aborting silently. What happened?

I don't understand why the update_migel_products_with_report needs to call a download_all of all regdata articles before iterating over all migel products. Also, why has this to be called several time. Doing it once for each language should suffice.

Modified it to skip this step and iterate over the konw migel codes. Waiting for update_migel_products_with_report to finish (Takes about 4 hours). Did not seen lines like last years import

2 / 568	Estimate total: 2.10 [h] It will be done in: 2.09 [h]
2015-02-02 18:10:40 +0100: save_all_products migel_code 01.01.02.01.2 count 2

Also remarked that update_products_by_migel_code checks the (outdated) pharmacode. This is not good. Do we need a rewrite part of the Migel code?

Fix NotImplementedError in rss update

Found the following error in import-daily

Plugin: ODDB::TextInfoPlugin
Error: NotImplementedError
Message: method `method_missing' called on terminated object (0x0000000820b2e8 flags=0x0 klass=0x0)
Backtrace:
/var/www/oddb.org/src/model/fachinfo.rb:91:in `localized_name'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/view/rss/fachinfo.rb:46:in `item_to_html'
/var/www/oddb.org/src/view/rss/fachinfo.rb:89:in `block (2 levels) in to_html'
/var/www/oddb.org/src/view/rss/fachinfo.rb:83:in `each'
/var/www/oddb.org/src/view/rss/fachinfo.rb:83:in `block in to_html'
/usr/local/lib/ruby/1.9.1/rss/maker/base.rb:438:in `make'
/usr/local/lib/ruby/1.9.1/rss/maker/base.rb:402:in `make'
/usr/local/lib/ruby/1.9.1/rss/maker.rb:9:in `make'
/var/www/oddb.org/src/view/rss/fachinfo.rb:74:in `to_html'
/var/www/oddb.org/src/plugin/plugin.rb:91:in `block (2 levels) in update_rss_feeds'
/var/www/oddb.org/src/plugin/plugin.rb:90:in `open'
/var/www/oddb.org/src/plugin/plugin.rb:90:in `block in update_rss_feeds'
/var/www/oddb.org/src/plugin/plugin.rb:49:in `call'
/var/www/oddb.org/src/plugin/plugin.rb:49:in `block in l10n_sessions'
/var/www/oddb.org/src/plugin/plugin.rb:46:in `each_key'
/var/www/oddb.org/src/plugin/plugin.rb:46:in `l10n_sessions'
/var/www/oddb.org/src/plugin/plugin.rb:82:in `update_rss_feeds'
/var/www/oddb.org/src/plugin/plugin.rb:77:in `block in update_yearly_fachinfo_feeds'
/var/www/oddb.org/src/plugin/plugin.rb:76:in `each'
/var/www/oddb.org/src/plugin/plugin.rb:76:in `update_yearly_fachinfo_feeds'
/var/www/oddb.org/src/plugin/text_info.rb:121:in `postprocess'
/var/www/oddb.org/src/plugin/text_info.rb:1490:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:573:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:549:in `call'
/var/www/oddb.org/src/util/updater.rb:549:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:567:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:344:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:198:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'

Pushed commit Corrected rescue. Show backtrace. Restarted import_daily on thinpower in screen import_daily.

Found the following (expected) lines in the screen

rss/fachinfo 68: rescue #<NotImplementedError: method `method_missing' called on unexpected T_NODE object (0x0000000a668cd0 flags=0x21c klass=0x0)> for rss/fachinfo 68: #<ODBA::Stub:738632360#31115107 @odba_class= @odba_container=745733800#21>: backtrace is: 
  /var/www/oddb.org/src/model/fachinfo.rb:91:in `localized_name'
  /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
  /var/www/oddb.org/src/view/rss/fachinfo.rb:46:in `item_to_html'
  /var/www/oddb.org/src/view/rss/fachinfo.rb:89:in `block (2 levels) in to_html'
  /var/www/oddb.org/src/view/rss/fachinfo.rb:83:in `each'
  /var/www/oddb.org/src/view/rss/fachinfo.rb:83:in `block in to_html'
Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Fach- und Patienteninfo Updates (swissmedicinfo.ch) - 01/2016 size 1497 with 0 attachments. 

Pat-Info completed successfully.

Remove unsed 'E'/12 from oddb_code.xml

Herr Sgier reported that the value '12', aka 'E', Kontraindiziert is never used. When adding support for EPHA http://dev.ywesee.com/Niklaus/20131126-integrate-epha-interaction they used 'E'. This was changed later, see http://dev.ywesee.com/Niklaus/20140331-add-e-option-to-oddb2xml where it changed to 'X'. The oddb2xml.xsd is correct and does not list a value of 12. But I forgot to remove it completely.

Pushed commits:

view · edit · sidebar · attach · print · history
Page last modified on January 12, 2016, at 03:16 PM