view · edit · sidebar · attach · print · history

Index>

20140811-Fix-invalid-byte-in-epha-interactions

Summary

  • Fix error invalid byte sequence in US-ASCII in ODDB::EphaInteractionPlugin
  • Add ZSR to prescription

Commits

(branch qrcode)

Index

Keep in Mind
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • One unit-test for searchbar fails and might be a clue why searching does not work correctly.
  • Added two skip in test/test_plugin/rss.rb. Why does the mocking not work there anymore?
  • Make ext/swissindex/test/test_swissindex.rb and skipping tests in migel-gem pass.

---

Add information accessible via ZSR to prescription

Idee is to make a JSON accessible interface to the ZSR information available from pharmed on our site.

Looked at Dojo documentation available under

Ruby json gem most recent version is 1.8.1. Okay.

Creating a src/util/zsr.rb with method ODDB::ZSR.info(zsr) which must return a valid information from the SOAP interface of pharmedsolution.ch. Unittest test/test_util/zsr.rb passes now using 'J039019' and 'J 0390.19'.

Must create now a src/view/zsr.rb which offers a JSON answer. Also added a test/test_view/zsr.rb. This can be tested with a watir tests accessing the URL base + 'zsr/<some_id>'.

But now I struggled already over an hour into trying to get oddb-ci2 to serve this URL. All requests just vanish and do not leave any trace. Adding some puts I see that src/util/process(request) is called using CGI. And then I see src/state/global/ state _trigger home. Forgot to change the src/util/validator. Now I am getting some small errors to correct.

Now the link de/gcc/zsr/id works as seen

Pushed my changes into branch qrcode Added json interface for ZSR.

Will continue tomorrow to fix the problem with the fachinfo search and integrating the ZSR to the QR-code.

Fix error invalid byte sequence in US-ASCII in ODDB::EphaInteractionPlugin

Looking at the content of the downloaded files as ruby 1.9 probably has an error recognizing UTF-8. The download from https://download.epha.ch/cleaned/matrix.csv is UTF-8 encoded and has (compared with the file from April 28) quotes around the field names and its content on each line. Looks like its format has changed again a little bit.

Message error was 

Plugin: ODDB::EphaInteractionPlugin
Error: ArgumentError
Message: invalid byte sequence in US-ASCII
Backtrace:
/var/www/oddb.org/src/plugin/epha_interactions.rb:40:in `match'
/var/www/oddb.org/src/plugin/epha_interactions.rb:40:in `block in update'
/var/www/oddb.org/src/plugin/epha_interactions.rb:37:in `each'
/var/www/oddb.org/src/plugin/epha_interactions.rb:37:in `update'
/var/www/oddb.org/src/util/updater.rb:527:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:503:in `call'
/var/www/oddb.org/src/util/updater.rb:503:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:521:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:337:in `update_epha_interactions'
/var/www/oddb.org/src/util/updater.rb:273:in `update_bsv_followers'
/var/www/oddb.org/src/util/updater.rb:204:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

Patched updater to run only update_epha_interactions when calling update_bsv_followers and testing the export on oddb-ci2 to see whether I can reproduce the error locally.

But locally I get an UTF-8 error earlier 

/var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: SL-Update (XML) - 08/2014 size Plugin: ODDB::BsvXmlPlugin
Error: REXML::ParseException
Message: #<Encoding::CompatibilityError: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)>
/usr/local/lib/ruby/1.9.1/rexml/source.rb:210:in `match'
/usr/local/lib/ruby/1.9.1/rexml/source.rb:210:in `match'
/usr/local/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:423:in `pull'
/usr/local/lib/ruby/1.9.1/rexml/parsers/streamparser.rb:16:in `parse'
/usr/local/lib/ruby/1.9.1/rexml/document.rb:205:in `parse_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:999:in `update_it_codes'
/var/www/oddb.org/src/plugin/bsv_xml.rb:660:in `block (2 levels) in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry.rb:500:in `get_input_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:660:in `block in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `block in each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/central_directory.rb:182:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:139:in `block in foreach'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:99:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:138:in `foreach'
/var/www/oddb.org/src/plugin/bsv_xml.rb:655:in `_update'
/var/www/oddb.org/src/plugin/bsv_xml.rb:650:in `update'
/var/www/oddb.org/src/util/updater.rb:247:in `block in update_bsv'
/var/www/oddb.org/src/util/updater.rb:503:in `call'
/var/www/oddb.org/src/util/updater.rb:503:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:245:in `update_bsv'
jobs/import_bsv:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_bsv:12:in `<module:Util>'
jobs/import_bsv:11:in `<module:ODDB>'
jobs/import_bsv:10:in `<main>'
...
Exception parsing
Line: -1
Position: -1
Last 80 unconsumed characters:
SISTEMA NERVOSO&#8198;</DescriptionIt>
Backtrace:
/usr/local/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:439:in `rescue in pull'
/usr/local/lib/ruby/1.9.1/rexml/parsers/baseparser.rb:330:in `pull'
/usr/local/lib/ruby/1.9.1/rexml/parsers/streamparser.rb:16:in `parse'
/usr/local/lib/ruby/1.9.1/rexml/document.rb:205:in `parse_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:999:in `update_it_codes'
/var/www/oddb.org/src/plugin/bsv_xml.rb:660:in `block (2 levels) in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry.rb:500:in `get_input_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:660:in `block in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `block in each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/central_directory.rb:182:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:139:in `block in foreach'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:99:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:138:in `foreach'
/var/www/oddb.org/src/plugin/bsv_xml.rb:655:in `_update'
/var/www/oddb.org/src/plugin/bsv_xml.rb:650:in `update'
/var/www/oddb.org/src/util/updater.rb:247:in `block in update_bsv'
/var/www/oddb.org/src/util/updater.rb:503:in `call'
/var/www/oddb.org/src/util/updater.rb:503:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:245:in `update_bsv'
jobs/import_bsv:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_bsv:12:in `<module:Util>'
jobs/import_bsv:11:in `<module:ODDB>'
jobs/import_bsv:10:in `<main>'
Non US-ASCII detected and no charset defined.
Defaulting to UTF-8, set your own if this is incorrect.

First rerun did not work as I forgot to reset the XMLPublications-latest.zip to state of July.

Now I get another error 

2014-08-11 09:12:16 CESTlog notify Error: SL-Update (XML): start outgoing process ["log"] 
2014-08-11 09:12:16 CEST /var/www/oddb.org/src/util/mail.rb: Configured email using /var/www/oddb.org/etc/oddb.yml @cfg is now "smtp.gmail.com" 587 "ngiger@ywesee.com"
2014-08-11 09:12:16 CESTUtil.send_mail list_and_recipients ["log"]
2014-08-11 09:12:16 CEST Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Error: SL-Update (XML) - 08/2014 size Plugin: ODDB::BsvXmlPlugin
Error: NoMethodError
Message: undefined method `configure' for Savon:Module
@report: {:name_base=>"Alfamino", :name_descr=>"Plv ", :swissmedic_no5_bag=>"00000", :deductible=>:deductible_g, :generic_type=>:unknown, :atc_class=>"V06DB", :pharmacode_bag=>"5339714"}
Backtrace:
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:34:in `initialize'
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:14:in `new'
(druby://localhost:50001) /var/www/oddb.org/ext/swissindex/src/swissindex.rb:14:in `session'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/invokemethod.rb:14:in `perform_with_block'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1506:in `perform'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1586:in `block (2 levels) in main_loop'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `loop'
(druby://localhost:50001) /usr/local/lib/ruby/1.9.1/drb/drb.rb:1582:in `block in main_loop'
/var/www/oddb.org/src/plugin/swissindex.rb:257:in `load_ikskey'
/var/www/oddb.org/src/plugin/bsv_xml.rb:244:in `load_ikskey'
/var/www/oddb.org/src/plugin/bsv_xml.rb:269:in `tag_start'
/usr/local/lib/ruby/1.9.1/rexml/parsers/streamparser.rb:24:in `parse'
/usr/local/lib/ruby/1.9.1/rexml/document.rb:205:in `parse_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:1007:in `update_preparations'
/var/www/oddb.org/src/plugin/bsv_xml.rb:660:in `block (2 levels) in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry.rb:500:in `get_input_stream'
/var/www/oddb.org/src/plugin/bsv_xml.rb:660:in `block in _update'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `call'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:42:in `block in each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/entry_set.rb:41:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/central_directory.rb:182:in `each'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:139:in `block in foreach'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:99:in `open'
/usr/local/lib/ruby/gems/1.9.1/gems/rubyzip-1.1.6/lib/zip/file.rb:138:in `foreach'
/var/www/oddb.org/src/plugin/bsv_xml.rb:655:in `_update'
/var/www/oddb.org/src/plugin/bsv_xml.rb:650:in `update'
/var/www/oddb.org/src/util/updater.rb:247:in `block in update_bsv'
/var/www/oddb.org/src/util/updater.rb:502:in `call'
/var/www/oddb.org/src/util/updater.rb:502:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:245:in `update_bsv'
jobs/import_bsv:14:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_bsv:12:in `<module:Util>'
jobs/import_bsv:11:in `<module:ODDB>'
jobs/import_bsv:10:in `<main>'
2014-08-11 09:12:18 CESTlog notify Error: SL-Update (XML): sent mail

Okay. I had a version 2.4.0 of savon installed, when I should have only 0.9.7.

Running it again I received an error message as I had an undefined get_mailing_list_anrede'. But luckily the update proceeded and I did see now in log/oddb/debug/2014/08.log the lines 

2014-08-11 12:49:58 +0200: EphaInteractionPlugin.update: /var/www/oddb.org/data/csv/interactions_de_utf8.csv ?  true
2014-08-11 12:49:58 +0200: deleting 2598 epha_interactions class Array
2014-08-11 12:50:10 +0200: Calling @app.odba_store
2014-08-11 12:56:55 +0200: Calling @app.epha_interactions.odba_store
2014-08-11 12:56:57 +0200: Calling @app.epha_interactions.each
2014-08-11 12:49:57 CEST getin update_bsv_followers                                       
2014-08-11 13:00:47 CESTlog notify Epha Interaktionen: start outgoing process ["log"]  
2014-08-11 13:00:47 CESTUtil.send_mail list_and_recipients ["log"]                                                                              
2014-08-11 13:00:47 CEST Util.log_and_deliver_mail to=["ngiger@ywesee.com"] subject ch.ODDB.org Report - Epha Interaktionen - 08/2014 size Read Epha                                                                                                                                                
2014-08-11 13:00:49 CESTlog notify Epha Interaktionen: sent mail                                                                                    
2014-08-11 13:00:49 CEST Done with update_epha_interactions

Committing changesForce input to utf-8 for bsv and epha_interactions and pushing them to to thinpower. Starting update_bsv_followers on thinpower in screen thread update_epha_interactions.

view · edit · sidebar · attach · print · history
Page last modified on August 11, 2014, at 05:02 PM