After running a FI import the newly created registration http://ch.oddb.org/de/gcc/fachinfo/reg/65731 cannot be found via search by trademark. Why? Looking via bin/admin I see
ch.oddb> registration('65731').name_base.class
-> NilClass
ch.oddb> registration('65731').sequences.keys
-> ["00"]
ch.oddb> registration('65731').sequences.values.first.name_base.class
-> NilClass
ch.oddb> registration('65731').packages.first
-> #<ODDB::Package:0x0000000a0959e8>
ch.oddb> registration('65731').packages.first.name_base.class
-> NilClass
ch.oddb> registration('65731').fachinfo.de.change_log.first.time
-> 2016-04-13
Trying to show the FI-diff via http://ch.oddb.org/de/gcc/fachinfo/reg/65731 results in TypeError can't convert nil into String. Looking via grep 65731 log/oddb/debug/2016/04.log produces
2016-04-13 07:06:32 +0200: /var/www/oddb.org/src/plugin/text_info.rb:615:in `create_registration': create_registration 65731/00/000 Nucala® company GlaxoSmithKline AG
2016-04-13 07:07:39 +0200: /var/www/oddb.org/src/plugin/text_info.rb:609:in `create_sequence': create_sequence 65731 seqNr 00 :!registration,65731!sequence,00. seq_args {:composition_text=>nil, :name_base=>"Nucala\u00AE", :name_descr=>nil, :dose=>nil, :sequence_date=>nil, :export_flag=>nil} registration.sequences {"00"=>}
2016-04-13 07:12:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:970:in `block (2 levels) in get_fi_pi_to_update': get_fi_pi_to_update: no reg or fi add #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode=nil, title="Nucala\u00AE", authHolder=nil, substances=nil, type="fi", lang="de", informationUpdate=nil, refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_de.xml", same_content_as_xml_file=nil>
2016-04-13 07:12:42 +0200: /var/www/oddb.org/src/plugin/text_info.rb:970:in `block (2 levels) in get_fi_pi_to_update': get_fi_pi_to_update: no reg or fi add #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode=nil, title="Nucala\u00AE", authHolder=nil, substances=nil, type="fi", lang="fr", informationUpdate=nil, refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_fr.xml", same_content_as_xml_file=nil>
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1175:in `extract_html': IKSNR 65731: File.size(/var/www/oddb.org/data/html/fi/fr/Nucala_.html) is 73519
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1234:in `parse_textinfo': must create textinfo for fi fr 65731 of ["65731"]
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:242:in `update_fachinfo_lang': update_fachinfo_lang 65731 #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode="R03DX09", title="Nucala\u00AE", authHolder="GlaxoSmithKline AG", substances="M\u00E9polizumab", type="fi", lang="fr", informationUpdate="01.2016", refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_fr.xml", same_content_as_xml_file=false>
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:253:in `update_fachinfo_lang': update_fachinfo_lang Nucala® iksnr 65731 store_fachinfo {} ["fr"] ATC R03DX09
2016-04-13 07:21:05 +0200: /var/www/oddb.org/src/plugin/text_info.rb:166:in `store_fachinfo': store_fachinfo: 65731 ["fr"] create_fachinfo :!fachinfo,34743651.
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:135:in `replace_textinfo': replace_textinfo updated :!registration,65731. type fachinfo
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:206:in `block in ensure_correct_atc_code': ensure_correct_atc_code iksnr 65731 save atcFromFI R03DX09 in sequence 65731 sequence 00 atc_class 34743675
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1175:in `extract_html': IKSNR 65731: File.size(/var/www/oddb.org/data/html/fi/de/Nucala_.html) is 65074
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:1234:in `parse_textinfo': must create textinfo for fi de 65731 of ["65731"]
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:242:in `update_fachinfo_lang': update_fachinfo_lang 65731 #<struct Struct::SwissmedicMetaInfo iksnr="65731", authNrs=["65731"], atcCode="R03DX09", title="Nucala\u00AE", authHolder="GlaxoSmithKline AG", substances="Mepolizumab", type="fi", lang="de", informationUpdate="01.2016", refdata=nil, xml_file="/var/www/oddb.org/data/details/65731_fi_de.xml", same_content_as_xml_file=false>
2016-04-13 07:21:07 +0200: /var/www/oddb.org/src/plugin/text_info.rb:253:in `update_fachinfo_lang': update_fachinfo_lang Nucala® iksnr 65731 store_fachinfo {} ["de"] ATC R03DX09
2016-04-13 07:21:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:156:in `store_fachinfo': store_fachinfo: 65731 ["de"] :!fachinfo,34743651. eql? false
2016-04-13 07:21:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:132:in `replace_textinfo': replace_textinfo updated :!registration,65731. :!registration,65731. type fachinfo :!fachinfo,34743651.
2016-04-13 07:21:08 +0200: /var/www/oddb.org/src/plugin/text_info.rb:194:in `ensure_correct_atc_code': ensure_correct_atc_code iksnr 65731 atcFromFI R03DX09 atcFromXml R03DX09 matched and found
IKSNR 65731: Nucala®
Fachinfo - DE - Nucala®["65731"]
Fachinfo - FR - Nucala®["65731"]
Fachinfo - DE - Nucala®["65731"]
Fachinfo - FR - Nucala®["65731"]
65731 ["fr"]
65731 ["de"]
After executing in bin/admin
ch.oddb> registration('65731').sequences.values.first.name_base = "Nucala®";
-> Nucala®
ch.oddb> registration('65731').sequences.values.first.odba_store;
-> Nucala®
ch.oddb>
I can find Nucala without any problems. But why did get the statement above not executed while importing the FI?
Looking at import of April 15 which add IKSNR 65304: BRONCHOSTOP® Hustenpastillen. It has the same problem, not exactly, as it has no FI but only a (skipped) PI as shown in the log 65304 BRONCHOSTOP® Hustenpastillen not found in Packungen.xlsx.
When I look at TextInfoPlugin::create_sequence in src/plugin/text_info.rb, the name_base of the sequence should be set correctly. But adding a line after creating the registration to enforce the same as logic as execute manually.
Running jobs/update_textinfo_swissmedicinfo --target=fi to see whether my idea works. But must first reload the db_dump.
First try did not work correctly. Also triggered some unit test errors. Pushed commit Update name_base when creating a sequence
Plugin: ODDB::TextInfoPlugin Error: NoMethodError Message: undefined method `packages' for "":String Backtrace: /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/model/registration.rb:241:in `block in packages' /var/www/oddb.org/src/model/registration.rb:240:in `each' /var/www/oddb.org/src/model/registration.rb:240:in `inject' /var/www/oddb.org/src/model/registration.rb:240:in `packages' /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' /var/www/oddb.org/src/plugin/text_info.rb:1194:in `get_textinfo' /var/www/oddb.org/src/plugin/text_info.rb:1451:in `block in import_swissmedicinfo' /var/www/oddb.org/src/plugin/text_info.rb:1449:in `each' /var/www/oddb.org/src/plugin/text_info.rb:1449:in `import_swissmedicinfo' /var/www/oddb.org/src/util/updater.rb:560:in `block in update_notify_simple' /var/www/oddb.org/src/util/updater.rb:536:in `call' /var/www/oddb.org/src/util/updater.rb:536:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:554:in `update_notify_simple' /var/www/oddb.org/src/util/updater.rb:333:in `update_textinfo_swissmedicinfo' /var/www/oddb.org/src/util/updater.rb:200:in `run' jobs/import_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `call' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_daily:12:in `<module:Util>' jobs/import_daily:11:in `<module:ODDB>' jobs/import_daily:10:in `<main>'
dmesg | grep memory on thinpower should e.g. this line [81361.182216] Out of memory: Kill process 11929 (ruby) score 1089 or sacrifice child and [ 3787.578024] Out of memory: Kill process 17396 (import_swissmed) score 924 or sacrifice child
I think we have problems, that some ch.oddb processes consume way too much memory and that the linux OOM kills one of them. This probably leads to the import-daily running forever.
Also ch.oddb is running very slow today. I suspect that some indices don't work correctly anymore. I think after an OOM of linux we should stop all oddb services, then stop postgres, restart postgres and all services to bring the system up again. Not necessary, as the oddb services got restarted automatically using the built-in resource limiting.
The stack trace show, that the call to packages comes after a call to TextInfoPlugin::create_sequence. Therefore it should not happen again, if we fix the first problem as we did with commit Update name_base when creating a sequence.
Must fix the error reported last week
ERROR /var/www/ydim.ywesee.com/doc/index.rbx:774: superclass mismatch for class Cookie /usr/lib64/ruby/1.8/cgi.rb:274:in `<top (required)>' /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require' /usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require' /var/www/ydim.ywesee.com/doc/index.rbx:4:in `<top (required)>' /usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `load' /usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `handler'
I banged my head, but I think it is a very bad (or almost impossible) idea to try to use one apache server for multiple rubies. See e.g. https://rvm.io/deployment/passenger3.
What could work however is that we run a second instance of apache for Ruby 1.8 application. If we must provide ydim.ywesee.com on the same port, we would proxy request to ydim.ywesee.com to the second instance of apache listening on a different port. This is documented under https://wiki.apache.org/httpd/RunningMultipleApacheInstances and /usr/share/doc/apache2/README.multiple-instances.
I also added the magic comment # encoding: utf-8to all ruby file under ydim-html. Also converted lib/ydim/html/util/lookandfeel.rb from ISO-8859-14 in ydim-htm/lib/ydim/html/util/lookandfeel.rb.
The I get in /var/log/apache2/error_log
ydim-html Client Error: Accept-Charset encoding error CGI::InvalidEncoding /usr/lib64/ruby/1.9.1/cgi/core.rb:669:in `block (2 levels) in initialize_query' /usr/lib64/ruby/1.9.1/cgi/core.rb:661:in `each' /usr/lib64/ruby/1.9.1/cgi/core.rb:661:in `block in initialize_query' /usr/lib64/ruby/1.9.1/cgi/core.rb:660:in `each' /usr/lib64/ruby/1.9.1/cgi/core.rb:660:in `initialize_query' /usr/lib64/ruby/1.9.1/cgi/core.rb:820:in `initialize' /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/request.rb:41:in `new' /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/request.rb:41:in `initialize' /var/www/ydim.ywesee.com/doc/index.rbx:16:in `new' /var/www/ydim.ywesee.com/doc/index.rbx:16:in `<top (required)>' /usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `load' /usr/lib64/ruby/1.9.1/apache/ruby-debug.rb:54:in `handler'
We see that the cgi from ruby 1.9.1 is called. Is it possible to encode everything correctly as UTF-8?
Adding some debug output in cgi/core.rb I get the following output in apaches error-log.
cgi CGI version "html4" cgi version UTF-8 664: key "debitor_type" val "dt_hosting" UTF-8 664: key "name" val "\xDCberreich" UTF-8 ydim-html Client Error: Accept-Charset encoding error CGI::InvalidEncoding
The String looks like an UTF-8 string. But the DRB ydim-html at the other side is running Ruby 1.8.6 which does not understand an UTF-8 string by default.
Conclusions
Resuming the changes made:
load File.expand_path('../../etc/config.rb', __FILE__) by load '/var/www/ydim.ywesee.com/etc/config.rb' to get rid of the Security error when using expand_path or load
require 'cgi' and require 'cgi/drbsession'
RubyAddPath /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib to be able load the sbsm/transhandler
if cuki = request.cookies[self::class::PERSISTENT_COOKIE_NAME]) by if false
pdfinvoice/invoice.rb:100: invalid multibyte char (US-ASCII) (SyntaxError). This could be easily fixed, but the underlying gem pdf-writer is stuck in version 1.1.8 from 2008 and does not work under ruby 1.9.
raise "Please, use ruby 1.9.0 or later." if RUBY_VERSION < "1.9.0"