20160531-sandoz-xmlconv
Summary
- Setup sandoz.xmlconv.bbmb.ch on oddb-ci2
Commits
Index
- Keep in Mind for work to do
- Fix dojo error http://www.sitepen.com/blog/2012/10/31/debugging-dojo-common-error-messages/#forgot-dom-ready
- I removed on May-27 tests for ix_registrationss, fix_sequences, fix_compositions, fix_packages from test/test_plugin/swissmedic.rb,as he could not find any references for them in the src code. Did I erroneously remove stuff when cleaning up the swissmedic import earlier?
- The whole test for older/newer Packages must be adapted to xlsx. One must compare the rows (e.g. by creating csv files) and do the same stuff in xlsx!
- creat gem: task: input=file with ean-codes, standard output show ean-codes + atc-code. Source is Swissmedic Packungen.xlsx or XML.
- Import via data/medreg_companies.yaml
- Fix problem with radioactivatum 99m-technetio when parsing Wirkstoffe
- Fix galenic_forms when parsing swissmedic.xlsx
- Cleanup generic_type. Replace it everywhere by sl_generic_type and adapt code accordingly.
- Get updated ATC-codes from EPha for oddb.org, too.
- Use refdatabase for oddb.org, too.
- Check whether we should revert the part which touche src/plugin/text_info.rb of commit 17af82ba4d76a5838683411b260de265531f9e74. We should improve test/stub/oddbapp.rb to work similar for update/pointer as the real oddbapp. In this case we would have a good Stub for plugins. May we need a different stub when working with plugins (which create/modify/destroy ODDB-Objects), when in most other cases a very simple stub is sufficient.
- When a logged in admin user changes an atc_code of a product, the corresponding atc_class must update its sequences, too.
- Order of entering search type and value should not matter. Both should show long URL with search
- Remove parser for minifi (but keep the minifi)
- BSV-Plugin does not send mail. Is it sending two mails at once?
- remove obsolete pdf_patinfo in model, admin/views
- Fix sbsm/htmlgrid when DEFAULT_FLAVOUR is nil
Setup sandoz.xmlconv.bbmb.ch on oddb-ci2
First must verify that I can log in as admin user as in http://sandoz.bbmb.ch/.
Dropped and reloaded the sandoz_yus db with the dump taken from fastpower. Took yus code from fastpower lke this:
- git clone git://scm.ywesee.com/yus/.git /usr/local/share/src/yus
- cd /usr/local/share/src/yus && git checkout c5f57e33a8967e9d6a9417e8af608e15215b74a4 -b fastpower
Now I am getting the following error
niklaus@oddb-ci2 /u/l/s/s/yus> sudo -u apache /usr/local/1.8.6-p388/bin/ruby18 -rubygems /usr/local/share/src/yus/bin/yusd config=/var/www/sandoz.bbmb.ch/etc/yus.yml
/usr/local/share/src/yus/lib/yus/server.rb:9: warning: already initialized constant VERSION
FEHLER: Relation »object« existiert bereits
FEHLER: Relation »prefetchable_index« existiert bereits
FEHLER: Relation »extent_index« existiert bereits
FEHLER: Relation »object_connection« existiert bereits
FEHLER: Relation »target_id_index« existiert bereits
FEHLER: Relation »collection« existiert bereits
/usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/marshal.rb:15:in `load': undefined class/module Encoding:: (ArgumentError)
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/marshal.rb:15:in `load'
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:629:in `restore'
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:319:in `fetch_or_restore'
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:314:in `call'
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:314:in `fetch_or_do'
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:318:in `fetch_or_restore'
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:66:in `bulk_restore'
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:63:in `each'
... 11 levels...
from /usr/local/1.8.6-p388/lib/ruby/site_ruby/1.8/odba/cache.rb:297:in `fetch_named'
from /usr/local/share/src/yus/lib/yus/persistence/odba.rb:14:in `initialize'
from /usr/local/share/src/yus/bin/yusd:53:in `new'
On fastpower I have
ngiger@fastpower /usr/local/share/src/yus $ find /usr/local/bin/ruby186 -name marshal.rb /usr/local/bin/ruby186/lib/ruby/1.8/soap/marshal.rb /usr/local/bin/ruby186/lib/ruby/1.8/xmlrpc/marshal.rb /usr/local/bin/ruby186/lib/ruby/gems/1.8/gems/odba-1.1.0/lib/odba/marshal.rb ngiger@fastpower /usr/local/share/src/yus $ sha256sum /usr/local/bin/ruby186/lib/ruby/gems/1.8/gems/odba-1.1.0/lib/odba/marshal.rb 7db1f320497ff4e288ea48e7f01a52fdc310619fe0af3f43b97dcf9d0e9715a5 /usr/local/bin/ruby186/lib/ruby/gems/1.8/gems/odba-1.1.0/lib/odba/marshal.rb
Copyied /usr/local/src/ruby-1.8.6-p369.tar.gz from thinpower and building it on oddb-ci2 (+ Rubygems 1.4.1) with this script: Attach:gen_ruby_186-p369.txt
Installed the gems with the following commands
sudo /usr/local/bin/ruby186/bin/gem install character-encodings --version=0.3.0 -- --with-cflags=\"-Wno-unused-but-set-variable\" sudo /usr/local/bin/ruby186/bin/gem install needle --version=1.3.0 sudo /usr/local/bin/ruby186/bin/gem install pg --version=0.8.0 sudo /usr/local/bin/ruby186/bin/gem install dbd-pg --version=0.3.9 sudo /usr/local/bin/ruby186/bin/gem install rclconf --version=1.0.0 sudo /usr/local/bin/ruby186/bin/gem install odba --version=1.1.0 sudo /usr/local/bin/ruby186/bin/gem install pg --version 0.8.0 sudo /usr/local/bin/ruby186/bin/gem install dbd-pg --version=0.3.9 sudo /usr/local/bin/ruby186/bin/gem install yus --version=1.0.1
Now yusd starts when colled like this sudo -u apache /usr/local/bin/ruby186/bin/ruby -rubygems /usr/local/bin/ruby186/bin/yusd config=/var/www/sandoz.bbmb.ch/etc/yus.yml
But on fastpower I have 4 different yusd instances running
ngiger@fastpower /var/www/sandoz.bbmb.ch $ ps -ef | grep yusd apache 2989 2975 0 2015 ? 00:33:39 /usr/local/bin/ruby186/bin/ruby -rubygems /usr/local/bin/ruby186/bin/yusd bbmb 3001 2997 0 2015 ? 1-01:19:19 /usr/local/bin/ruby186/bin/ruby -rubygems /usr/local/share/src/yus/bin/yusd config=/var/www/sandoz.bbmb.ch//etc/yus.yml bbmb 3022 3019 0 2015 ? 22:53:41 /usr/local/bin/ruby186/bin/ruby -rubygems /usr/local/share/src/yus/bin/yusd config=/var/www/vetoquinol.bbmb.ch//etc/yus.yml bbmb 3034 3028 0 2015 ? 1-12:14:13 /usr/local/bin/ruby186/bin/ruby -rubygems /usr/local/share/src/yus/bin/yusd config=/var/www/virbac.bbmb.ch//etc/yus.yml
The situation in /usr/local/share/src/yus is the following. Checkout is
ngiger@fastpower /var/www/sandoz.bbmb.ch $ git log -1
commit c5f57e33a8967e9d6a9417e8af608e15215b74a4
Author: Yasuhiro Asaka <grauwoelfchen@gmail.com>
Date: Sat Jun 15 02:26:07 2013 +0900
Rename pg driver name to Pg
- pg(ruby-pg) gem expects 'Pg' (not 'pg')
ngiger@fastpower /usr/local/share/src/yus $ git status -uno
# On branch master
nothing to commit (use -u to show untracked files)
ngiger@fastpower /usr/local/share/src/yus $ cat .git/config
[core]
repositoryformatversion = 0
filemode = true
bare = false
logallrefupdates = true
[remote "origin"]
url = git://scm.ywesee.com/yus/.git
fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
remote = origin
merge = refs/heads/master
Checkout out the same commit as branch fastpower on /usr/local/src/yus and now /service/sandoz.bbmb.ch-yus/ starts up correctly.
A succesfull login on http://sandoz.bbmb.ch produces entries like ths
==> /service/sandoz.bbmb.ch-yus/log/main/current <== @40000000574d43ce2739fc0c I, [2016-05-31T09:56:52.657976 #3001] INFO -- Yus::Server: Login attempt for xxx@sandoz.ch from ch.bbmb.sandoz @40000000574d43ce273b28d4 I, [2016-05-31T09:56:52.658155 #3001] INFO -- Yus::Server: Authentication succeeded for xxx@sandoz.ch @40000000574d43ce27a6620c D, [2016-05-31T09:56:52.665172 #3001] DEBUG -- Yus::EntitySession: allowed?(login, ch.bbmb.sandoz.Admin) ==> /service/sandoz.bbmb.ch/log/main/current <== @40000000574d43ce27b398dc D, [2016-05-31T09:56:52.665946 #11429] DEBUG -- User: xxx@sandoz.ch: allowed?(login, ch.bbmb.sandoz.Admin) -> true ==> /service/sandoz.bbmb.ch-yus/log/main/current <== @40000000574d43ce27bbe9c4 D, [2016-05-31T09:56:52.666560 #3001] DEBUG -- Yus::EntitySession: allowed?(login, ch.bbmb.sandoz.Customer) ==> /service/sandoz.bbmb.ch/log/main/current <== @40000000574d43ce27e9376c D, [2016-05-31T09:56:52.669520 #11429] DEBUG -- User: admin@sandoz.ch: allowed?(login, ch.bbmb.sandoz.Customer) -> false @40000000574d43cf1054afdc D, [2016-05-31T09:56:53.273843 #11429] DEBUG -- State: Customers#init: loaded 1661 customers in 0.60362s
Where as on oddb-ci2 it fails like this
==> /service/sandoz.bbmb.ch-yus/log/main/current <== @40000000574d40cf39159814 I, [2016-05-31T09:44:05.957637 #6779] INFO -- Yus::Server: Login attempt for xxx@sandoz.ch from ch.bbmb @40000000574d40cf3917035c I, [2016-05-31T09:44:05.957775 #6779] INFO -- Yus::Server: Authentication succeeded for xxx@sandoz.ch @40000000574d40cf3a06d1fc D, [2016-05-31T09:44:05.973448 #6779] DEBUG -- Yus::EntitySession: allowed?(login, ch.bbmb.Admin) ==> /service/sandoz.bbmb.ch/log/main/current <== @40000000574d40cf3a0e731c D, [2016-05-31T09:44:05.973949 #7956] DEBUG -- User: xxx@sandoz.ch: allowed?(login, ch.bbmb.Admin) -> false
It looks as if some msiconfiguratioin. On fastpower bbmb looks for ch.bbmb.sandoz.Admin whereas here it looks for ch.bbmb.Admin. Found it. I uncommented the corresponding line for the auth_domain in /var/www/sandoz.bbmb.ch/etc/config.yml. But now I get a [2016-05-31T10:07:30.792452 #6779] WARN -- Yus::Server: Authentication failed for admin@sandoz.ch Addes user bbmb to group postgres. And now I find Yus::Server: Authentication succeeded for xxx@sandoz.ch. But the Apache configuration is still wrong I see mod_ruby: failed to require sbsm/trans_handler. Made this problem going away by correcting the RubyAddPath. Now I can login via http://sandoz.oddb-ci2.dyndns.org/de/customers/, but after the login I get the error ERR_TOO_MANY_REDIRECTS. And in the /var/www/sandoz.bbmb.ch/log/access_log I see many entries like 172.25.1.75 - - [31/May/2016:11:18:49 +0200] "GET /de/customers/ HTTP/1.1" 302 - "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.36 Safari/537.36"
Also in the /service/sandoz.bbmb.ch-yus/log/main/current I see several occurrences of Server: Authentication succeeded for xxx@sandoz.ch. What is happening? Looks that we have a problem handling the states in bbmb, as lib/bbmb/html/state/customers.rb is responsible for printing State: Customers#init: loaded 1661 customers in 0.60362s
Apache config looks this Attach:apache_sandoz_bbmb_ch.txt
Simplyfied the apache_sandoz_bbmb_ch.conf to retain only the relevant stuff of sandoz.bbmb.ch. Tried changing the servername to sandoz.localhost. This did not work. Changed it to sandoz.oddb-ci2.dyndns.org and it work again.
Changed to RubyAddPath /usr/local/lib/ruby/gems/2.3.0/gems/sbsm-1.2.7/lib/sbsm/trans_handler.rb. Now I am getting the error
[Tue May 31 14:37:18 2016] [error] mod_ruby: error in ruby [Tue May 31 14:37:18 2016] [error] mod_ruby: /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:62:in `file?': Insecure operation - file? (SecurityError) [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:62:in `block (2 levels) in contains_requirable_file?' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:62:in `each' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:62:in `any?' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:62:in `block in contains_requirable_file?' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:60:in `each' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:60:in `any?' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/basic_specification.rb:60:in `contains_requirable_file?' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:925:in `block in find_inactive_by_path' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:924:in `each' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:924:in `find' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/specification.rb:924:in `find_inactive_by_path' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems.rb:185:in `try_activate' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:132:in `rescue in require' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:144:in `require' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /var/www/sandoz.bbmb.ch/doc/index.rbx:3:in `<top (required)>' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/1.9.1/apache/ruby-run.rb:53:in `load' [Tue May 31 14:37:18 2016] [error] mod_ruby: from /usr/local/lib/ruby/1.9.1/apache/ruby-run.rb:53:in `handler' [Tue May 31 14:37:18 2016] [error] [client 172.25.1.75] File does not exist: /var/www/sandoz.bbmb.ch/doc/favicon.ico, referer: http://sandoz.oddb-ci2.dyndns.org/
Attach:apache_sandoz_bbmb_ch2.txt
Trying to fix the error using untaint did not help in April when I ported ydim to Ruby 2.3.0. The RubyAddPath directive must only include the modules used for the mod_ruby (running under Ruby 1.9 in our case). Then a DRB session must be opened. Therefor I compare the ydim/doc/index.rbx with the sandoz.bbmb.ch/doc/index.rb. The Apache configuration differs because ydim uses https, and sandoz.bbmb.ch uses http.
Trying to replace "DRb.start_service('druby://localhost:0')" with "::DRb.start_service('druby://localhost:0')" in the doc/index.rbx. (I already had removed the require 'encoding/character/utf-8'
Reverting to use the apache_sandoz_bbmb_ch.txt. Slowly uncommenting lines to see what breaks loading the the doc/index.rbx. Only difference now is
RubyAddPath /usr/local/lib64/ruby/gems/1.9.1/gems/sbsm-1.2.3/lib instead of
RubyAddPath /usr/local/lib/ruby/gems/2.3.0/gems/sbsm-1.2.7/lib/sbsm/trans_handler.rb
Stil having the question why in /service/sandoz.bbmb.ch-yus/log/main/current I see after each login 8 times
@40000000574d98ad087a95ec D, [2016-05-31T15:58:59.142194 #18462] DEBUG -- User: xxx@sandoz.ch: allowed?(login, ch.bbmb.sandoz.Admin) -> true @40000000574d98ad0886461c D, [2016-05-31T15:58:59.142981 #18462] DEBUG -- User: xx@sandoz.ch: allowed?(login, ch.bbmb.sandoz.Customer) -> false @40000000574d98ad147cf114 D, [2016-05-31T15:58:59.343660 #18462] DEBUG -- State: Customers#init: loaded 1661 customers in 0.20024s
When looking at etc/ydim.yml I see the etc/ydim.yml file which specified a YDIM-URL. Adapting it to oddb-ci2 and trying to start the /service/ydim before starting sandoz.bbmb.ch. The services ydim and ydim-html are running and I can the the port 12375 using
niklaus@oddb-ci2 /v/w/sandoz.bbmb.ch> sudo netstat -lptu | grep 12375 tcp6 0 0 oddb-ci2.ngiger.d:12375 [::]:* LISTEN 29359/bin/ydimd
Remarked that http://oddb-ci2.dyndns.org/ displays the same stuff as http://sandoz.oddb-ci2.dyndns.org/.
Found a /usr/local/share/src/yus/bin/yusd config=/var/www/sandoz.bbmb.ch/etc/yus.yml belonging to the apache user, which was not killed correctly this morning. Killed it with -9. Shutting down /service/yus to ensure that it does not answer any request from sandoz.
Renamed the Servername to sandoz.bbmb.ngiger.ch. Removed duplicated LoadModule ruby_module in /etc/apache/vhost.d/*.conf files.
Tasks for tomorrow are:
- Do we need to grant yus entities as described by Mase in http://dev.ywesee.com/Masa/20101110-setup-eshop?
- Copy sandoz.xmlconf files from fastpower, generate a git repository and try to convert it into a gem
- Fix skipped unit tests in bbmb
- Try to make the selenium test run under bbmb. If successfull also in sandoz.bbmb.ch and sandoz.xmlconf