Must merge the changes from the ruby2x branch into master, then adapt the apache, crontab & services entries to the new convention on thin power. Backup existiting config files to /home/ywesee/oddb.1.93.
As on oddb-ci2 I find in /etc/apache2/vhosts.d/oddb.conf only entries for RubyAddPath like
/etc/apache2/vhosts.d/oddb.conf: RubyAddPath /var/www/oddb.org/src /etc/apache2/vhosts.d/oddb.conf: RubyAddPath /usr/local/bin/ruby186/lib/ruby/gems/1.8/gems/sbsm-1.2.6/lib /etc/apache2/vhosts.d/oddb.conf: RubyAddPath /usr/local/bin/ruby186/lib/ruby/gems/1.8/gems/rclconf-1.0.0/lib
I do not touch any apache.conf on thinpower (But I think maybe we will have problems because of the missing rclconf-1.0.0/lib.
The following services run files must be patched:
/service/ch.oddb-crawler/run:cd /var/www/oddb.org /service/ch.oddb-exporter/run:exec sudo -u apache /usr/local/bin/ruby /var/www/oddb.org/ext/export/bin/exportd /service/ch.oddb-fiparse/run:exec sudo -u apache /usr/local/bin/ruby193 /var/www/oddb.org/ext/fiparse/bin/fiparsed /service/ch.oddb-google_crawler/run:cd /var/www/oddb.org /service/ch.oddb-meddata/run:exec sudo -u apache /usr/local/bin/ruby /var/www/oddb.org/ext/meddata/bin/meddatad /service/ch.oddb-readonly/run:exec sudo -u apache /usr/local/bin/ruby193 -I /var/www/de.oddb.org/lib /var/www/oddb.org/ext/readonly/bin/readonlyd /service/ch.oddb-refdata/run:exec sudo -u apache /usr/local/bin/ruby /var/www/oddb.org/ext/refdata/bin/refdatad /service/ch.oddb-swissindex_nonpharma/run:exec sudo -u apache /usr/local/bin/ruby /var/www/oddb.org/ext/swissindex/bin/swissindexd /service/ch.oddb-swissregd/run:exec sudo -u apache /usr/local/bin/ruby /var/www/oddb.org/ext/swissreg/bin/swissregd /service/ch.oddb/run:cd /var/www/oddb.org
First running /usr/local/bin/bundle-240 bundle install as user ywesee in /var/www/oddb.org. Checking whether sudo -u apache bundle-240 exec ruby-240 bin/oddbd can find all needed libraries and services.
First modified /service/ch.oddb-crawler/run. Restart was okay. No fixed /services/ch-oddb/run. Restarted and did a quick check. FI and searchin price for isacador worked. Working on the other services
New commands in services/run will look like exec sudo -u apache bundle-240 exec ruby-240 bin/oddbd
Had to to deactivate the following services
Pushed commit Removed unused require archive/tarsimple and oddb_yaml/exporter need syck engine. This fixed starting ch.oddb-exporter.
Pushed commit Removed support for parsing fachinfo PDFs. This fixed starting /service/ch.oddb-fiparse/.
Remarked that the google crawler often restarts after approximately 40 seconds of uptime. Investigating.
Found in /service/ch.oddb-crawler/log/main/current the line
@40000000587c877a1f4cee8c Footprint exceeds 1450MB. Exiting status_crawler.
Increased in src/util/oddbapp.rb the limit from 1460 to MEMORY_LIMIT_CRAWLER = 2048. This seems to be okay. htop reports between 1200 and 1700 MB of used memory for OddbApp:Crawler. Pushed commit Updated memory limit for crawler
Still found restarts for crawler with 2048 MB. Added a timestamp to the log entry in the hope to be able to see what causes the problem. Increased (not commited) limit to 2.5 GB.
Looking at all the error_log under /var/www/oddb.org/log/*/. Funny entries are in log/mobile/error_log, log/phyto-pharma/error_log and
[Mon Jan 16 09:13:57 2017] [error] [client 109.128.78.142] File does not exist: /var/www/oddb.org/doc/apple-touch-icon-precomposed.png [Mon Jan 16 09:13:57 2017] [error] [client 109.128.78.142] File does not exist: /var/www/oddb.org/doc/apple-touch-icon.png [Mon Jan 16 09:41:14 2017] [error] [client 91.200.12.73] File does not exist: /var/www/oddb.org/doc/wp-admin, referer: http://oddb.org/wp-admin/css/colors-classic.css
Found
[Mon Jan 16 10:03:35 2017] [error] mod_ruby: error in ruby [Mon Jan 16 10:03:35 2017] [error] mod_ruby: (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:375:in `_id2ref': 0x3fc8b87a6168 is not id value (RangeError) [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:375:in `to_obj' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/site_ruby/1.8/odba/drbwrapper.rb:70:in `to_obj' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:1403:in `to_obj' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:1705:in `to_obj' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:613:in `recv_request' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:911:in `recv_request' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:1531:in `init_with_client' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from (druby://localhost:11000) /usr/lib64/ruby/1.8/drb/drb.rb:1543:in `setup_message' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: ... 26 levels... [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:55:in `require' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/trans_handler.rb:126:in `uri_parser' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/trans_handler.rb:75:in `parse_uri' [Mon Jan 16 10:03:35 2017] [error] mod_ruby: from /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib/sbsm/trans_handler.rb:106:in `translate_uri' error_log [1] RO 1-1, 72 of 118, (9014) Ende 0x5B (91) $
The proces from port 11000 belongs to de.oddb.org. Wondering why de.oddb.org calls directly ch.oddb.org. Ignoring it for the moment.
Also I cannot explain the error from phyto-pharma. But I do not think it is related to ruby 2.4.0
Updated /etc/crontab to use bundler. Rebuilding all indices in screen bin_admin.
Investigating the problem. Added a pry in src/view/drugs/resultlist.rb:254
[62] pry(#<ODDB::View::Drugs::ResultList>)> model.atc_facades.first.packages.first.sequence.composition_text => "extractum aquosum liquidum fermentatum 0.05 mg ex viscum album (abietis) recens 0.01 mg, natrii chloridum, aqua q.s. ad solutionem pro 1 ml." [63] pry(#<ODDB::View::Drugs::ResultList>)> model.atc_facades.first.packages.first.sequence.pointer => #<ODDB::Persistence::Pointer:0x007f87c005cce8 @directions=[[:registration, "56830"], [:sequence, "01"]]> [75] pry(#<ODDB::View::Drugs::ResultList>)> model.atc_facades.first.packages.first.sequence.packages.values.last.swissmedic_source[:import_date] => #<Date: 2016-05-12 ((2457521j,0s,0n),+0s,2299161j)> [64] pry(#<ODDB::View::Drugs::ResultList>)> x.sequence.pointer => #<ODDB::Persistence::Pointer:0x007f87c00883e8 @directions=[[:registration, "56829"], [:sequence, "23"]]> [65] pry(#<ODDB::View::Drugs::ResultList>)> x.sequence.composition_text => "extractum aquosum liquidum fermentatum 0.05 mg ex viscum album (mali) recens 0.01 mg, natrii chloridum, argenti carbonas (0,01 ug pro 100 mg herba recente), aqua q.s. ad solutionem pro 1 ml." [73] pry(#<ODDB::View::Drugs::ResultList>)> x.sequence.packages.values.last.swissmedic_source[:import_date] => #<Date: 2016-05-12 ((2457521j,0s,0n),+0s,2299161j)>
Verified with bin/admin that the package which is displayed without an active_agents was imported in 2015, the one with an active agent was imported in 2016
ch.oddb> registration('56829').package('158').swissmedic_source[:import_date].to_s
-> 2016-05-12
ch.oddb> registration('56829').package('158').active_agents.size
-> 1
ch.oddb> registration('56830').package('001').swissmedic_source[:import_date].to_s
-> 2015-02-09
ch.oddb> registration('56830').package('001').composition_text
-> extractum aquosum liquidum fermentatum 0.05 mg ex viscum album (abietis) recens 0.01 mg, natrii chloridum, aqua q.s. ad solutionem pro 1 ml.
ch.oddb> registration('56830').package('001').active_agents.size
-> 0
Remarked that the import_date is not updated when a new FI is parsed. Is this okay, eg. for 58763 which was imported today we still have in bin/admin
ch.oddb> registration('58763').sequence('001').packages.values.find_all { |pac| pac.swissmedic_source }[0].swissmedic_source[:import_date]
-> 2014-12-04
ch.oddb> registration('58763').sequence('001').packages.values.find_all { |pac| pac.swissmedic_source }[1].swissmedic_source[:import_date]
-> 2014-12-04
Modified the check parameter jobs/import_swissmedic_only to recheck every package. Not only if the package is absent.
Looking again at the composition_test I see that 56829 and 56830 are really different 56830 has no active agenten, wherease 56829 has 56829 "Argenti Carbonas (0,01 ug pro 100 mg Herba Recente)" or maybe I should recognise "ex viscum album (abietis)" extract (Weißbeerigen Mistel).see http://de.wikipedia.org/wiki/Wei%C3%9Fbeerige_Mistel
ch.oddb> registration('56829').package('158').composition_text
-> extractum aquosum liquidum fermentatum 0.05 mg ex viscum album (mali) recens 0.01 mg, natrii chloridum, argenti carbonas (0,01 ug pro 100 mg herba recente), aqua q.s. ad solutionem pro 1 ml.
ch.oddb> registration('56830').package('001').composition_text
-> extractum aquosum liquidum fermentatum 0.05 mg ex viscum album (abietis) recens 0.01 mg, natrii chloridum, aqua q.s. ad solutionem pro 1 ml.
ch.oddb> registration('56830').package('001').atc_class.descriptions
-> {"en"=>"Other Plant Alkaloids and Natural Products"}
ch.oddb> registration('56829').package('158').atc_class.descriptions
-> {"en"=>"Other Plant Alkaloids and Natural Products"}
ch.oddb> registration('56829').package('158').atc_class.code
-> L01CX
ch.oddb> registration('56830').package('001').atc_class.code
-> L01CX
ch.oddb> registration('56829').package('158').active_agents.size
-> 1
ch.oddb> registration('56829').package('158').active_agents.first
-> Argenti Carbonas (0,01 ug pro 100 mg Herba Recente)
In the Packungen.xls we have 342 occurrences of 'ex'. Followed or not by ','. Examples are
extractum ethanolicum ex chelidonii radix recens 22.5 mg polysaccharida ex streptococcus pneumoniae (serotypes: 1, 2, 3, 4, 5, 6B, 7F, 8, 9N, 9V, 10A, 11A, 12F, 14, 15B, 17F, 18C, 19A, 19F, 20, 22F, 23F, 33F) je 25 µg, natrii chloridum limonis succus 8-12 mg, extractum aquosum 30 mg ex cydoniae fructus recens 9.7 mg, ad solutionem pro 1 ml. flavonoidea 200 mg ex limonis flavedo recens, hesperidini methylchalconum 40 mg, esculosidum 10 mg, excipiens pro capsula. extractum spissum ex: echinaceae purpureae herbae recentis tinctura 380 mg, ratio: 1:12, et echinaceae purpureae radicis recentis tinctura 20 mg, ratio: 1:11, excipiens pro compresso. II) Fettemulsion: sojae oleum 50 g, triglycerida saturata media 50 g, lecithinum ex ovo 6 g, glycerolum, natrii oleas, aqua q.s. ad emulsionem pro 500 ml. extractum aquosum liquidum 20 mg ex viscum album (pini) recens 1 mg, natrii chloridum, aqua q.s. ad solutionem pro 1 ml.
Iscador is already in the oddb2xml spec test where we find
context 'find correct result compositions for 56829 sequence 3 Iscador M 0,01 mg' do
comment_from_email_good_7_juni_2011 = %(
Ausgedeutscht heisst das:
Der Extrakt ist ein Auszug aus der Frischpflanze im Verhältnis 1:5, also extractum 0.05 mg entspricht 0.01 mg frischem Mistelkraut.
Es handelt sich um EINEN Wirkstoff, also in Kombination ist falsch formuliert.
Die HILFSSTOFFE sind Aqua ad iniectabilia und Natrii chloridum.
)
<..>
skip { expect(viscum.is_active_agent).to eq true } # TODO: Grenzfall. Active-Agent viscum album (mali) recens ist sub-string
If we want to fix this problem we should add a fluid form 'Extract' (and may be also a similar for solid) as I think this is also possible.
Using bundle-240 exec ruby-240 bin/admin shows
ch.oddb> search_oddb('Novartis', 'de').package_count
-> 398
ch.oddb> search_oddb('mepha', 'de').package_count
-> 199
ch.oddb> search_oddb('sandoz', 'de').package_count
-> 6093
Do we have problem when diplaying very long lists/arrays?
We have a limit to display over 50 sequences when logged in as administrator, which display :w_too_many_sequences => 'Zu viele Suchresultate, die Suche muss spezifiziert werden'. But this is not the case here.
Set a pry debug point at /var/www/oddb.org/src/state/global.rb:892. Pry snippet is
[1] pry(#<ODDB::State::Drugs::Result>)> res = @session.search_exact_company(query); ODBA::OdbaResultLimitError: ODBA::OdbaResultLimitError from /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:468:in `retrieve_from_index'
Added another breakpoint at /var/www/oddb.org/vendor/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:468.
This is trigger when a method error_limit returns a lower result than allowed. In src/util/oddb_app.rb we find RESULT_SIZE_LIMIT = 250. In the case of sandoz we found 256 elements. The index used was atc_index_company.
After doubling RESULT_SIZE_LIMIT to 500 I was able to display Liste für "Sandoz" (1216)Sortierung nach Stärke, Packung und Preis. But why did we get another result with Ruby 2.4.0 than with 1.9.3?
Rebuilding the index using sudo -u apache bundle-240 exec ruby-240 jobs/rebuild_indices atc_index_company to see whether afterwards we still have the same problem. Yes indeed. This solved the problem. Rebuilding the same index on thinpower.
Rebuilding all indices on thinpower to check, whether we can to this without generating any error.
The daily drug cost for dafalgan kinder is wrong (1.40) as displayed by http://ch.oddb.org/de/gcc/ddd_price/reg/51231/seq/03/pack/031/search_query/ben-u-ron/search_type/st_oddb. The information displayed is
Tageskosten für Dafalgan Kinder Tagesdosis 3 g Publikumspreis 1.40 CHF Stärke 250 mg Packungsgrösse 12 Sachet(s) Berechnung ( 3 g / 250 mg ) x ( 1.40 / 12 Sachet(s) ) = 1.40 CHF / Tag
The dosis as displayed is http://ch.oddb.org/de/gcc/search/zone/drugs/search_query/51231/search_type/st_registration paracetamolum 250 mg, aromatica, aspartamum, conserv.: E 211, excipiens ad pulverem pro charta.. ATC-Klassierung is Paracetamol (N02BE01). Oddb has the following information about it
N02BE01 - Paracetamol Stärke Adm.Route Note 3 g R 3 g P 3 g O
Verified against https://www.whocc.no/atc_ddd_index/?code=N02BE01 which shows
TC code Name DDD U Adm.R Note N02BE01 paracetamol 3 g O 3 g P 3 g Rand defines the following abbreviation
g = gram O = oral P = parenteral R = rectal
It is a pouder, therefore O. 3 g DDD correspond to 12 x 250mg ą 1.40, which should give a daily cost of 1.40.
The ddd_price is calculated in src/model/package.rb. Also there are already two tests for ddd_price in test/test_model/package.rb. Adding a new one for dafalgan.
My first try displayed the following error
TestPackage#test_ddd_daflagan_kinder [test/test_model/package.rb:358]: Expected: "16.80" Actual: "1.68"
Somehow a factor was not converted correctly. There is probably an error in quanty. Looks like quanty was once a Ruby-Gem. Quanty is now just some files under src/util/quanty.
Looked at https://github.com/olbrich/ruby-units which I thing should be useful. Here my irb session
irb(main):032:0> mg250 = Unit.new('250 mg').base
=> 1/4000 kg
irb(main):033:0> g3 = Unit.new('3 g').base
=> 3/1000 kg
irb(main):034:0> g3/mg250
=> 12
=> RubyUnits::Unit
irb(main):036:0> (g3/mg250).to_i
=> 12
irb(main):037:0> (g3/mg250).to_i.class
=> Integer
irb(main):009:0> m = Unit.new('100 µg') ; m2= Unit.new('2 h')
=> 2 h
irb(main):010:0> m/m2
=> 50 ug/h
irb(main):002:0> Unit.new('100 µg/2 h')
=> 50 ug/h
irb(main):003:0> Unit.new('100 µg/ 2 h')
ArgumentError: '100 µg/ 2 h' Unit not recognized
from /home/niklaus/git/oddb.org/vendor/ruby240/ruby/2.4.0/gems/ruby-units-2.1.0/lib/ruby_units/unit.rb:1553:in `parse'
from /home/niklaus/git/oddb.org/vendor/ruby240/ruby/2.4.0/gems/ruby-units-2.1.0/lib/ruby_units/unit.rb:519:in `initialize'
from (irb):3:in `new'
from (irb):3
from /home/niklaus/.rbenv/versions/2.4.0/bin/irb:11:in `<top (required)>'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `load'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:74:in `kernel_load'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/cli/exec.rb:27:in `run'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:332:in `exec'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor.rb:359:in `dispatch'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:20:in `dispatch'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/vendor/thor/lib/thor/base.rb:440:in `start'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/cli.rb:11:in `start'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/exe/bundle:34:in `block in <top (required)>'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/lib/bundler/friendly_errors.rb:100:in `with_friendly_errors'
from /home/niklaus/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/bundler-1.13.7/exe/bundle:26:in `<top (required)>'
from /home/niklaus/.rbenv/versions/2.4.0/bin/bundle:22:in `load'
from /home/niklaus/.rbenv/versions/2.4.0/bin/bundle:22:in `<main>'
The last example works with ODDB::Dose.new(100, 'µg / 2 h') and returns correctly Quanty(50,'µg/h').
Therefore I will look tomorrow whether I can easily swap Quanty by ruby-gem while preserving compatiblity with the stored odba values.