view · edit · sidebar · attach · print · history

Index>

20140916-fix-analsis-search

Summary

  • Fix analyses search. Eg. for Akute myeloische Leukämie

Commits

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!

---

Fix analyses search. Eg. for Akute myeloische Leukämie

Trying to understand the problem via bin/admin.

Integrate update of analysis list into daily update

The BAG publishes also the list of the medical analysis under http://www.bag.admin.ch/themen/krankenversicherung/00263/00264/04185/index.html?lang=de where we find links like http://www.bag.admin.ch/themen/krankenversicherung/00263/00264/04185/index.html?lang=de&download=NHzLpZeg7t,lnp6I0NTU042l2Z6ln1acy4Zn4Z2qZpnO2Yuq2Z6gpJCLeoJ4gWym162epYbg2c_JjKbNoKSn6A-- for the XLS-data file.

The short link http://www.bag.admin.ch/al redirects to the above mentioned page and I think it will be more stable. The update happens only once or twice a year, but the file file is small (around 100KB). Therefore we download it every day and check for changes.

Running jobs/update_analysis on oddb-ci2. Job take much time and I do not see any progress. Remarked that no unit test exists for it. Creating one.

jobs/update_analysis died with

disabling UPDATER
/var/www/oddb.org/log/job.pid is created
process: jobs/update_analysis
init system
init system: 11.139719801
setup drb-delegation
/var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: Analysis - 09/2014 size Plugin: ODDB::AnalysisPlugin
Error: Ole::Storage::FormatError
Message: OLE2 signature is invalid
Backtrace:
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.7/lib/ole/storage/base.rb:378:in `validate!'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.7/lib/ole/storage/base.rb:370:in `initialize'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.7/lib/ole/storage/base.rb:112:in `new'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.7/lib/ole/storage/base.rb:112:in `load'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.7/lib/ole/storage/base.rb:79:in `initialize'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.7/lib/ole/storage/base.rb:85:in `new'
/usr/local/lib/ruby/gems/1.9.1/gems/ruby-ole-1.2.11.7/lib/ole/storage/base.rb:85:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/spreadsheet-0.9.7/lib/spreadsheet/excel/reader.rb:1276:in `setup'
/usr/local/lib/ruby/gems/1.9.1/gems/spreadsheet-0.9.7/lib/spreadsheet/excel/reader.rb:134:in `read'
/usr/local/lib/ruby/gems/1.9.1/gems/spreadsheet-0.9.7/lib/spreadsheet/excel/workbook.rb:32:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/swissmedic-diff-0.1.8/lib/compatibility.rb:13:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/swissmedic-diff-0.1.8/lib/compatibility.rb:19:in `open'
/var/www/oddb.org/src/plugin/analysis.rb:74:in `parse_xls'
/var/www/oddb.org/src/plugin/analysis.rb:40:in `update_group_position'
/var/www/oddb.org/src/plugin/analysis.rb:33:in `update'
/var/www/oddb.org/src/util/updater.rb:216:in `block in update_analysis'
/var/www/oddb.org/src/util/updater.rb:496:in `call'
/var/www/oddb.org/src/util/updater.rb:496:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:214:in `update_analysis'
jobs/update_analysis:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/update_analysis:12:in `<module:Util>'
jobs/update_analysis:11:in `<module:ODDB>'
jobs/update_analysis:10:in `<main>'
/var/www/oddb.org/log/job.pid is deleted

This probably explains why we have no analysis data as the first step when calling update is delete_all_analysis_group. Problem is that it is now also an XLSX and no more an XLS file.

Job completed fine. bin/admin shows now

 analysis_groups.size
-> 19
 analysis_positions.size
-> 50
ch.oddb> analysis_groups.first
-> ["1", #<ODBA::Stub:93295080#32263605 @odba_class=ODDB::Analysis::Group @odba_container=93303900#954379>]
 analysis_positions.first.taxpoints[0..80]
-> Supplément de transition, par analyse, pour des analyses effectuées au laboratoir
 analysis_positions.first.localized_name(:de)
-> 
 analysis_positions.first.search_text('de')
-> undefined method `split' for nil:NilClass

As analysis_index_de has resolve_search_term: 'search_text(:de)' we have problems. I also wonder why we have only 50 analysis_positions. On thinpower I found

ch.oddb> analysis_positions.size
-> 1669
ch.oddb> analysis_groups.size
-> 1250
ch.oddb> analysis_positions.first.search_text(:de)
-> 125Dihydroxycholecalciferol 1000 100000

Rebuilding the indices sudo -u apache /usr/local/bin/ruby jobs/rebuild_indices analysis_index_de to be sure.

Searching in ch.oddb.org under analysis for 125Dihydroxycholecalciferol works. But not on oddb-ci2. Therefore not yet correctly parsing the xlsx-file. Adding more debug-info and improving the unit test.

view · edit · sidebar · attach · print · history
Page last modified on September 16, 2014, at 10:56 AM