could not connect to www.swissreg.ch: #<Net::HTTPInternalServerError:0x007f8a7d69bb58>
---
Pushed some commits with stuff forgotten in previous tasks:
See also problem of last wednesday. Trying to ensure / rescue this error on oddb-ci2. Added also debug statement in storage.rb
Now when start bin/oddbd the following output
init system
ensure_object_connections: ensure 2 target_ids [689832, 26257256, 10, 6, 934654, 795431, 9, 11, 868627, 3, 20, 18, 8, 724033, 1252741, 726317, 17, 16, 13, 393389, 780360, 19, 1211301, 15, 5, 12, 7, 724030, 14, 1067430, 9587859, 4, 22, 21, 810961, 954379, 28724410, 28834017, 28850745, 30791407] sql SELECT target_id FROM object_connection
WHERE origin_id = ?
init system: 5.665840982
Now running jobs/import_daily on oddb-ci2.
To understand better the difference between the old (and good) way of running bin/oddbd and the new one /bin/bash -c cd /var/www/oddb.org && bundle exec /usr/local/bin/ruby jobs/import_daily I must dig into the source code of bundler.
In lib/bundler/cli.rb we see that the exec method does the following stuff
def exec(*args)
Bundler.definition.validate_ruby! # Validates that the ruby and gem version work together
Bundler.load.setup_environment # in lib/bundler/runtime.rb
<..>
# Run
Kernel.exec(*args)
<..>
end
The setup_environment method manipulates the environments variables BUNDLE_BIN_PATH, PATH, BUNDLE_GEMFILE, RUBYOPT und RUBYLIB ( to restrict loading gems. When looking at http://bundler.io/v1.3/rationale.html we see, that we did not yet the lines
require 'rubygems' require 'bundler/setup'
to ensure that oddb.org uses bundler. This was verified (by coincidence today). When I got an DRB-Connection error while trying to run import_daily in a screen and the code was using a drb.rb not in inside vendor/bundle. See this stack
Plugin: ODDB::TextInfoPlugin Error: DRb::DRbConnError Message: druby://localhost:10002 - #<Errno::ECONNREFUSED: Connection refused - connect(2)> Backtrace: /usr/local/lib/ruby/1.9.1/drb/drb.rb:736:in `rescue in block in open' /usr/local/lib/ruby/1.9.1/drb/drb.rb:730:in `block in open' /usr/local/lib/ruby/1.9.1/drb/drb.rb:729:in `each' /usr/local/lib/ruby/1.9.1/drb/drb.rb:729:in `open' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1191:in `initialize' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1171:in `new' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1171:in `open' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1087:in `block in method_missing' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1105:in `with_friend' /usr/local/lib/ruby/1.9.1/drb/drb.rb:1086:in `method_missing' /var/www/oddb.org/src/plugin/text_info.rb:109:in `parse_patinfo' /var/www/oddb.org/src/plugin/text_info.rb:1112:in `block in parse_and_update' /var/www/oddb.org/src/plugin/text_info.rb:1076:in `each' <..>
Added the required lines to jobs/import_daily, bin/oddbd, bin/admin and ext/fiparse/bin/fiparsed and rerunning it. Now I found the reason, why my fiparsed did not run. It was trying to load rwv2/rwv which is no longer in Gemfile. After commenting out the line require 'fachinfo_doc' in ext/fiparse/src/fiparse.rb the fiparsed started without problems and import_daily completed its work (found 0 rescue and 1549 ensure entries). Attached the file vendor/bundle/ruby/1.9.1/gems/odba-1.1.0/lib/odba/storage.rb as Attach:storage.txt
To complete the migration to use bundler, I recommend the following steps
require 'bundler/setup' to all files under bin, jobs, ext/*/bin
These services are found like this
grep apache /service/ch.od*/run | grep -v bundle /service/ch.oddb-fiparse/run:exec sudo -u apache /usr/local/bin/ruby193 /var/www/oddb.org/ext/fiparse/bin/fiparsed /service/ch.oddb-meddata/run:exec sudo -u apache /usr/local/bin/ruby193 /var/www/oddb.org/ext/meddata/bin/meddatad /service/ch.oddb-readonly/run:exec sudo -u apache /usr/local/bin/ruby193 -I /var/www/de.oddb.org/lib /var/www/oddb.org/ext/readonly/bin/readonlyd /service/ch.oddb-swissindex_nonpharma/run:exec sudo -u apache /usr/local/bin/ruby193 /var/www/oddb.org/ext/swissindex/bin/swissindex_nonpharmad /service/ch.oddb-swissindex_pharma/run:exec sudo -u apache /usr/local/bin/ruby193 /var/www/oddb.org/ext/swissindex/bin/swissindex_pharmad /service/ch.oddb-swissregd/run:exec sudo -u apache /usr/local/bin/ruby193 /var/www/oddb.org/ext/swissreg/bin/swissregd
As this solution is too invasive (e.g. we must modify over 30 ruby scripts) Zeno any myself agreed on the following solution
Looking at the Gemfile.lock I found it not easy to parse and handling stuff like handling the https://github.com/zdavatz/ruby-dbi.git. Therefore gave a closer look to the possible use of bundler and I think bundler should to right stuff if I call it like
sudo bundle --system --shebang /usr/local/bin/ruby --without=debugger Fetching gem metadata from http://rubygems.org/....... Fetching gem metadata from http://rubygems.org/....... Fetching additional metadata from http://rubygems.org/.. Fetching additional metadata from http://rubygems.org/.. Fetching https://github.com/zdavatz/ruby-dbi.git Klone in Bare-Repository '/usr/local/lib64/ruby/gems/1.9.1/cache/bundler/git/ruby-dbi-18dc939ce8385c18d6bc000c645b5b9ad027a87e'... remote: Reusing existing pack: 3609, done. remote: Counting objects: 21, done. remote: Compressing objects: 100% (21/21), done. remote: Total 3630 (delta 5), reused 10 (delta 0) Empfange Objekte: 100% (3630/3630), 1.14 MiB | 718.00 KiB/s, done. Löse Unterschiede auf: 100% (1638/1638), done. Prüfe Konnektivität... Fertig. Using rake (10.1.0) <..> Using bundler (1.5.2) Your bundle is complete! Gems in the group debugger were not installed. Use `bundle show [gemname]` to see where a bundled gem is installed. Post-install message from money: Please note the following API changes in Money version 6 - Money#amount, Money#dollars methods now return instances of BigDecimal (rather than Float). Please read the migration notes at https://github.com/RubyMoney/money#migration-notes and choose the migration that best suits your application. Test responsibly :-)
Testing it on oddb-ci2 gives
sudo -u apache bin/oddbd 2>&1 | tee /mnt/oddb.org/oddbd_storage_bundler_system.log
/usr/lib64/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:53:in `require': iconv will be deprecated in the future, use String#encode instead.
/usr/lib64/ruby/site_ruby/1.9.1/rubygems/specification.rb:2007:in `raise_if_conflicts': Unable to activate mechanize-2.7.3, because mime-types-1.25 conflicts with mime-types (~> 2.0) (Gem::LoadError)
from /usr/lib64/ruby/site_ruby/1.9.1/rubygems/specification.rb:1176:in `activate'
from /usr/lib64/ruby/site_ruby/1.9.1/rubygems.rb:186:in `rescue in try_activate'
<...>
After trying quite a few alternatives I think these commands should be okay
#!/bin/bash -v set -e sudo /usr/local/bin/gem install bundler sudo /usr/local/bin/gem update --system sudo -u apache /usr/local/bin/bundle install --path /var/www/.gem/
Afterwards one can launch the jobs/binaries using a fully qualified path for ruby sudo -u apache /usr/local/bin/ruby bin/oddbd. The advantage of this approch is that /var/www/.gem/ is currently non existent. On thinpower we must assure that the user apache has write access to /var/www/oddb.org/.bundle/config.
Set /etc/gemrc to gem: --no-document to avoid installing ri and rdoc. Found out that I must first install wasabi and then savon when using gem.
Removed all installed gems in oddb-ci2 with the following helper
gems =`gem list`.split("\n")
gems.each {
|gem|
next unless m= /^(\w*)/.match(gem)
cmd = "gem uninstall --executables --force --all #{m[1]}"
puts cmd
system(cmd)
}
Created install_gems.sh, see commit install gems with a Bash-Script not with Bundler After discussion with Zeno I could remove some more gems.Added gems needed for development/test as comment. See commit Corrections. Added gems for development
Changed puppet receipts with commit Run services without bundler. Relaunched all services. Running oddbd and migeld with a log file. Replaced data/xls/Packungen-latest.xls with an old file. Started im/export_daily again.
Results:
Message: could not connect to www.swissreg.ch: #<Net::HTTPInternalServerError:0x007fbe14383fe0>
Error: NoMethodError Message: undefined method `type' for nil:NilClass Backtrace: (druby://localhost:10005) /var/www/oddb.org/src/model/doctor.rb:46:in `block in praxis_address' (druby://localhost:10005) /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `each' (druby://localhost:10005) /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `find' (druby://localhost:10005) /usr/local/lib64/ruby/gems/1.9.1/gems/odba-1.1.0/lib/odba/stub.rb:112:in `method_missing' (druby://localhost:10005) /var/www/oddb.org/src/model/doctor.rb:45:in `praxis_address' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/csv_exporter.rb:103:in `first_address_data' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/csv_exporter.rb:77:in `block in collect_data' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/csv_exporter.rb:67:in `collect' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/csv_exporter.rb:67:in `collect_data' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/csv_exporter.rb:99:in `block in dump' (druby://localhost:10005) /usr/lib64/ruby/1.9.1/csv.rb:1367:in `open' (druby://localhost:10005) /var/www/oddb.org/ext/export/src/csv_exporter.rb:98:in `dump'
I am not sure whether the 37 uniq lines containg unable to replace are a real problem or might even be related to the fact that we no longer support fachinfo_doc. The lines were
ODBA::Stub was unable to replace ODDB::SimpleLanguage::Descriptions#30593148 from ODDB::Fachinfo:#30593146 ODBA::Stub was unable to replace ODDB::Text::Chapter#31059255 from ODDB::FachinfoDocument2001:#31059249 ODBA::Stub was unable to replace ODDB::Text::Chapter#31059256 from ODDB::FachinfoDocument2001:#31059249 ODBA::Stub was unable to replace ODDB::Text::Chapter#31059259 from ODDB::FachinfoDocument2001:#31059249 ODBA::Stub was unable to replace ODDB::Text::Chapter#31059266 from ODDB::FachinfoDocument2001:#31059249 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074232 from ODDB::FachinfoDocument2001:#31074226 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074233 from ODDB::FachinfoDocument2001:#31074226 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074236 from ODDB::FachinfoDocument2001:#31074226 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074243 from ODDB::FachinfoDocument2001:#31074226 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074696 from ODDB::FachinfoDocument2001:#31074690 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074697 from ODDB::FachinfoDocument2001:#31074690 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074700 from ODDB::FachinfoDocument2001:#31074690 ODBA::Stub was unable to replace ODDB::Text::Chapter#31074707 from ODDB::FachinfoDocument2001:#31074690 ODBA::Stub was unable to replace ODDB::Text::Chapter#31084838 from ODDB::FachinfoDocument2001:#31084832 ODBA::Stub was unable to replace ODDB::Text::Chapter#31084839 from ODDB::FachinfoDocument2001:#31084832 ODBA::Stub was unable to replace ODDB::Text::Chapter#31084842 from ODDB::FachinfoDocument2001:#31084832 ODBA::Stub was unable to replace ODDB::Text::Chapter#31084849 from ODDB::FachinfoDocument2001:#31084832 ODBA::Stub was unable to replace ODDB::Text::Chapter#31095013 from ODDB::FachinfoDocument2001:#31095007 ODBA::Stub was unable to replace ODDB::Text::Chapter#31095014 from ODDB::FachinfoDocument2001:#31095007 ODBA::Stub was unable to replace ODDB::Text::Chapter#31095017 from ODDB::FachinfoDocument2001:#31095007 ODBA::Stub was unable to replace ODDB::Text::Chapter#31095024 from ODDB::FachinfoDocument2001:#31095007 ODBA::Stub was unable to replace ODDB::Text::Chapter#31098713 from ODDB::FachinfoDocument2001:#31098707 ODBA::Stub was unable to replace ODDB::Text::Chapter#31098714 from ODDB::FachinfoDocument2001:#31098707 ODBA::Stub was unable to replace ODDB::Text::Chapter#31098717 from ODDB::FachinfoDocument2001:#31098707 ODBA::Stub was unable to replace ODDB::Text::Chapter#31098724 from ODDB::FachinfoDocument2001:#31098707 ODBA::Stub was unable to replace ODDB::Text::Chapter#31103781 from ODDB::FachinfoDocument2001:#31103775 ODBA::Stub was unable to replace ODDB::Text::Chapter#31103782 from ODDB::FachinfoDocument2001:#31103775 ODBA::Stub was unable to replace ODDB::Text::Chapter#31103785 from ODDB::FachinfoDocument2001:#31103775 ODBA::Stub was unable to replace ODDB::Text::Chapter#31103792 from ODDB::FachinfoDocument2001:#31103775 ODBA::Stub was unable to replace ODDB::Text::Chapter#31109236 from ODDB::FachinfoDocument2001:#31109230 ODBA::Stub was unable to replace ODDB::Text::Chapter#31109237 from ODDB::FachinfoDocument2001:#31109230 ODBA::Stub was unable to replace ODDB::Text::Chapter#31109240 from ODDB::FachinfoDocument2001:#31109230 ODBA::Stub was unable to replace ODDB::Text::Chapter#31109247 from ODDB::FachinfoDocument2001:#31109230 ODBA::Stub was unable to replace ODDB::Text::Chapter#31115440 from ODDB::FachinfoDocument2001:#31115434 ODBA::Stub was unable to replace ODDB::Text::Chapter#31115441 from ODDB::FachinfoDocument2001:#31115434 ODBA::Stub was unable to replace ODDB::Text::Chapter#31115444 from ODDB::FachinfoDocument2001:#31115434 ODBA::Stub was unable to replace ODDB::Text::Chapter#31115451 from ODDB::FachinfoDocument2001:#31115434
See commit * Revert "Avoid SSL-problem with ruby 1.9.3p0".
Actions on thinpower were:
sudo curl -O http://curl.haxx.se/ca/cacert.pem mv cacert.pem /etc/ssl/cert.pem