view · edit · sidebar · attach · print · history

< Niklaus.20160419-ydim-html-ruby-1-9 | Index | Niklaus.20160414-ydim-html >>

20160418-ydim-html

Summary

  • Fix error in YDIM html
  • Fix undefined method `packages' for "":String in Patinfo-Import
  • New Fachinfo must be found using search_by_trademark

Commits

Index

Keep in Mind for work to do
  • Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
  • I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
  • The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
  • creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
  • Import via data/medreg_companies.yaml
  • Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
  • Fix galenic_forms when parsing swissmedic.xlsx
  • Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
  • Get updated ATC-codes from EPha for oddb.org, too.
  • Use refdatabase for oddb.org, too.
  • Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
  • When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
  • Order of entering search type and value should not matter. Both should show long URL with search
  • Remove parser for minifi (but keep the minifi)
  • BSV-Plugin does not send mail. Is it sending two mails at once?
  • remove obsolete pdf_patinfo in model, admin/views

New Fachinfo must be found using search_by_trademark

After running a FI import the newly created registration http://ch.oddb.org/de/gcc/fachinfo/reg/65731 cannot be found via search by trademark. Why? Looking via bin/admin I see

ch.oddb> registration('65731').name_base.class
-> NilClass
ch.oddb> registration('65731').sequences.keys
-> ["00"]
ch.oddb> registration('65731').sequences.values.first.name_base.class
-> NilClass
ch.oddb> registration('65731').packages.first
-> #<ODDB::Package:0x0000000a0959e8>
ch.oddb> registration('65731').packages.first.name_base.class
-> NilClass
ch.oddb> registration('65731').fachinfo.de.change_log.first.time
-> 2016-04-13

Trying to show the FI-diff via http://ch.oddb.org/de/gcc/fachinfo/reg/65731 results in TypeError can't convert nil into String. Looking via grep 65731 log/oddb/debug/2016/04.log produces

2016-04-13 07:06:32 +0200: /var/www/oddb.org/src/plugin/text_info.rb:615:in `create_registration': create_registration 65731/00/000 Nucala® company GlaxoSmithKline AG
2016-04-13 07:07:39 +0200: /var/www/oddb.org/src/plugin/text_info.rb:609:in `create_sequence': create_sequence 65731 seqNr 00  :!registration,65731!sequence,00. seq_args {:composition_text=>nil, :name_base=>"Nucala\u00AE", :name_descr=>nil, :dose=>nil, :sequence_date=>nil, :export_flag=>nil} registration.sequences {"00"=>}
2016-04-13 07:12:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:970:in `block (2 levels) in get_fi_pi_to_update': get_fi_pi_to_update: no reg or fi add #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode=nil, title="Nucala\u00AE", authHolder=nil, substances=nil, type="fi", lang="de", informationUpdate=nil, refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_de.xml", same_content_as_xml_file=nil>
2016-04-13 07:12:42 +0200: /var/www/oddb.org/src/plugin/text_info.rb:970:in `block (2 levels) in get_fi_pi_to_update': get_fi_pi_to_update: no reg or fi add #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode=nil, title="Nucala\u00AE", authHolder=nil, substances=nil, type="fi", lang="fr", informationUpdate=nil, refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_fr.xml", same_content_as_xml_file=nil>
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1175:in `extract_html': IKSNR 65731: File.size(/var/www/oddb.org/data/html/fi/fr/Nucala_.html) is 73519
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1234:in `parse_textinfo': must create textinfo for fi fr 65731 of ["65731"]
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:242:in `update_fachinfo_lang': update_fachinfo_lang 65731 #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode="R03DX09", title="Nucala\u00AE", authHolder="GlaxoSmithKline AG", substances="M\u00E9polizumab", type="fi", lang="fr", informationUpdate="01.2016", refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_fr.xml", same_content_as_xml_file=false>
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:253:in `update_fachinfo_lang': update_fachinfo_lang Nucala® iksnr 65731 store_fachinfo {} ["fr"] ATC R03DX09
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:166:in `store_fachinfo': store_fachinfo: 65731 ["fr"] create_fachinfo :!fachinfo,34743651.
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:135:in `replace_textinfo': replace_textinfo updated :!registration,65731. type fachinfo
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:206:in `block in ensure_correct_atc_code': ensure_correct_atc_code iksnr 65731 save atcFromFI R03DX09 in sequence 65731 sequence 00 atc_class  34743675
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1175:in `extract_html': IKSNR 65731: File.size(/var/www/oddb.org/data/html/fi/de/Nucala_.html) is 65074
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1234:in `parse_textinfo': must create textinfo for fi de 65731 of ["65731"]
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:242:in `update_fachinfo_lang': update_fachinfo_lang 65731 #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode="R03DX09", title="Nucala\u00AE", authHolder="GlaxoSmithKline AG", substances="Mepolizumab", type="fi", lang="de", informationUpdate="01.2016", refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_de.xml", same_content_as_xml_file=false>
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:253:in `update_fachinfo_lang': update_fachinfo_lang Nucala® iksnr 65731 store_fachinfo {} ["de"] ATC R03DX09
2016-04-13 07:21:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:156:in `store_fachinfo': store_fachinfo: 65731 ["de"] :!fachinfo,34743651. eql? false
2016-04-13 07:21:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:132:in `replace_textinfo': replace_textinfo updated :!registration,65731. :!registration,65731. type fachinfo :!fachinfo,34743651.
2016-04-13 07:21:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:194:in `ensure_correct_atc_code': ensure_correct_atc_code iksnr 65731 atcFromFI R03DX09 atcFromXml R03DX09 matched and found
  IKSNR 65731: Nucala® 
  Fachinfo - DE - Nucala®["65731"]
  Fachinfo - FR - Nucala®["65731"]
  Fachinfo - DE - Nucala®["65731"]
  Fachinfo - FR - Nucala®["65731"]
  65731 ["fr"] 
  65731 ["de"] 

After executing in bin/admin

ch.oddb> registration('65731').sequences.values.first.name_base = "Nucala®";
-> Nucala®
ch.oddb> registration('65731').sequences.values.first.odba_store;
-> Nucala®
ch.oddb> 

I can find Nucala without any problems. But why did get the statement above not executed while importing the FI?

Looking at import of April 15 which add IKSNR 65304: BRONCHOSTOP® Hustenpastillen. It has the same problem, not exactly, as it has no FI but only a (skipped) PI as shown in the log 65304 BRONCHOSTOP® Hustenpastillen not found in Packungen.xlsx.

When I look at TextInfoPlugin::create_sequence in src/plugin/text_info.rb, the name_base of the sequence should be set correctly. But adding a line after creating the registration to enforce the same as logic as execute manually.

Running jobs/update_textinfo_swissmedicinfo --target=fi to see whether my idea works. But must first reload the db_dump.

First try did not work correctly. Also triggered some unit test errors. Pushed commit Update name_base when creating a sequence

Fix undefined method `packages' for "":String in Patinfo-Import

Plugin: ODDB::TextInfoPlugin
Error: NoMethodError
Message: undefined method `packages' for "":String
Backtrace:
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/model/registration.rb:241:in `block in packages'
/var/www/oddb.org/src/model/registration.rb:240:in `each'
/var/www/oddb.org/src/model/registration.rb:240:in `inject'
/var/www/oddb.org/src/model/registration.rb:240:in `packages'
/usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing'
/var/www/oddb.org/src/plugin/text_info.rb:1194:in `get_textinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1451:in `block in import_swissmedicinfo'
/var/www/oddb.org/src/plugin/text_info.rb:1449:in `each'
/var/www/oddb.org/src/plugin/text_info.rb:1449:in `import_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:560:in `block in update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:536:in `call'
/var/www/oddb.org/src/util/updater.rb:536:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:554:in `update_notify_simple'
/var/www/oddb.org/src/util/updater.rb:333:in `update_textinfo_swissmedicinfo'
/var/www/oddb.org/src/util/updater.rb:200:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `call'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

dmesg | grep memory on thinpower should e.g. this line [81361.182216] Out of memory: Kill process 11929 (ruby) score 1089 or sacrifice child and [ 3787.578024] Out of memory: Kill process 17396 (import_swissmed) score 924 or sacrifice child

I think we have problems, that some ch.oddb processes consume way too much memory and that the linux OOM kills one of them. This probably leads to the import-daily running forever.

Also ch.oddb is running very slow today. I suspect that some indices don't work correctly anymore. I think after an OOM of linux we should stop all oddb services, then stop postgres, restart postgres and all services to bring the system up again. Not necessary, as the oddb services got restarted automatically using the built-in resource limiting.

The stack trace show, that the call to packages comes after a call to TextInfoPlugin::create_sequence. Therefore it should not happen again, if we fix the first problem as we did with commit Update name_base when creating a sequence.

Fix error in YDIM html

Must fix the error reported last week

ERROR
/var/www/ydim.ywesee.com/doc/index.rbx:774: superclass mismatch for class Cookie
/usr/lib64/ruby/1.8/cgi.rb:274:in `<top (required)>'
/usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
/usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require'
/var/www/ydim.ywesee.com/doc/index.rbx:4:in `<top (required)>'
/usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `load'
/usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `handler'

I banged my head, but I think it is a very bad (or almost impossible) idea to try to use one apache server for multiple rubies. See e.g. https://rvm.io/deployment/passenger3.

What could work however is that we run a second instance of apache for Ruby 1.8 application. If we must provide ydim.ywesee.com on the same port, we would proxy request to ydim.ywesee.com to the second instance of apache listening on a different port. This is documented under https://wiki.apache.org/httpd/RunningMultipleApacheInstances and /usr/share/doc/apache2/README.multiple-instances.

I also added the magic comment # encoding: utf-8to all ruby file under ydim-html. Also converted lib/ydim/html/util/lookandfeel.rb from ISO-8859-14 in ydim-htm/lib/ydim/html/util/lookandfeel.rb.

The I get in /var/log/apache2/error_log

ydim-html Client Error: Accept-Charset encoding error
CGI::InvalidEncoding
/usr/lib64/ruby/1.9.1/cgi/core.rb:669:in `block (2 levels) in initialize_query'
/usr/lib64/ruby/1.9.1/cgi/core.rb:661:in `each'
/usr/lib64/ruby/1.9.1/cgi/core.rb:661:in `block in initialize_query'
/usr/lib64/ruby/1.9.1/cgi/core.rb:660:in `each'
/usr/lib64/ruby/1.9.1/cgi/core.rb:660:in `initialize_query'
/usr/lib64/ruby/1.9.1/cgi/core.rb:820:in `initialize'
/usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/request.rb:41:in `new'
/usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/request.rb:41:in `initialize'
/var/www/ydim.ywesee.com/doc/index.rbx:16:in `new'
/var/www/ydim.ywesee.com/doc/index.rbx:16:in `<top (required)>'
/usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `load'
/usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `handler'

We see that the cgi from ruby 1.9.1 is called. Is it possible to encode everything correctly as UTF-8?

Adding some debug output in cgi/core.rb I get the following output in apaches error-log.

cgi CGI version "html4"
cgi version UTF-8
664: key "debitor_type" val "dt_hosting" UTF-8
664: key "name" val "\xDCberreich" UTF-8
ydim-html Client Error: Accept-Charset encoding error
CGI::InvalidEncoding

The String looks like an UTF-8 string. But the DRB ydim-html at the other side is running Ruby 1.8.6 which does not understand an UTF-8 string by default.

Conclusions

Resuming the changes made:

  1. in ydim-html/doc/index.rbx replace load File.expand_path('../../etc/config.rb', __FILE__) by load '/var/www/ydim.ywesee.com/etc/config.rb' to get rid of the Security error when using expand_path or load
  2. in lib/sbsm/request.rb uncomment require 'cgi' and require 'cgi/drbsession'
  3. in /etc/apache2/vhosts.d/oddb.conf add RubyAddPath /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib to be able load the sbsm/transhandler
  4. in /usr/lib64/ruby/site_ruby/1.8/sbsm/session.rb replace if cuki = request.cookies[self::class::PERSISTENT_COOKIE_NAME]) by if false
  5. ydim does not work under 1.9 as pdfinvoice throws errors like pdfinvoice/invoice.rb:100: invalid multibyte char (US-ASCII) (SyntaxError). This could be easily fixed, but the underlying gem pdf-writer is stuck in version 1.1.8 from 2008 and does not work under ruby 1.9.
  6. The version of cgi.rb between 1.8 and 1.9.1 are incompatible, e.g. look at line 13 of /usr/lib64/ruby/1.9.1/cgi.rb which says raise "Please, use ruby 1.9.0 or later." if RUBY_VERSION < "1.9.0"
  7. as we try to pass ruby 1.9.1 CGI request to a DRB process ydim-html running under Ruby 1.8 this fails.
  • Trying to force UTF-8 encoding in sbsm/request.rb:process did not work.
  • Trying to force UTF-8 encoding in lib/htmlgrid/component.rb _to_html did not work.
view · edit · sidebar · attach · print · history
Page last modified on April 18, 2016, at 06:10 PM