Today I want to analyse the logs of oddb.org to check them for any irregularities and will be looking for searches and/or other accesses where the user has to wait too long.
My idea is to create a simple ruby helper script, which reads in a log file and converts it to a sqlite database where it is easy to filter/analyse the results.
Calling gem install data_mapper dm-sqlite-adapter.
Using curl -v --user-agent 'Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)' http://generika.oddb-ci2.dyndns.org/fr/generika/search/zone/drugs/search_query/Metaneuron+N/search_type/st_oddb/currency/EUR. The access is still logged in /var/www/oddb.org.rack/log/2017/07/11/generika_log
Checking apache config. The following lines must be place in each virtual host of the apache conf
# Rewrite rule for bots on port 8512 -> 8112 or 8222
RewriteMap lc int:tolower
RewriteCond %{HTTP_USER_AGENT} "google"
RewriteRule ^/(.*)$ http://localhost:8112/$1 [P,L]
RewriteCond %{HTTP_USER_AGENT} "archiver|slurp|bot|crawler|jeeves|spider|\.{6}"
RewriteRule ^/(.*)$ http://localhost:8212/$1 [P,L]
before redirecting to the specific port via a line like RewriteRule ^/(.*)$ http://localhost:8512/$1 [P]
Installed and activated new apache conf on thinpower.
This seems to work as the following commands (and visiting the given url in the browser)
curl -v --user-agent 'Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)' http://generika.cc/fr/generika/search/zone/drugs/search_query/Metaneuron+N/search_type/st_oddb/currency/EUR curl -v --user-agent 'Mozilla/5.0 (compatible; google-crawler' http://generika.cc/fr/generika/search/zone/drugs/search_query/Metaneuron+N/search_type/st_oddb/currency/EUR
lead to the this log entries
grep Metaneuron /var/www/oddb.org.rack/log/2017/07/11/*log /var/www/oddb.org.rack/log/2017/07/11/crawler_log:80.218.53.88 - - [11/Jul/2017 09:36:19] "GET http://generika.cc/fr/generika/search/zone/drugs/search_query/Metaneuron+N/search_type/st_oddb/currency/EUR HTTP/1.1" 200 356858 1.6167 "Mozilla/5.0 (compatible; SemrushBot/1.2~bl; +http://www.semrush.com/bot.html)" /var/www/oddb.org.rack/log/2017/07/11/generika_log:80.218.53.88 - - [11/Jul/2017 09:41:58] "GET http://generika.cc/fr/generika/search/zone/drugs/search_query/Metaneuron+N/search_type/st_oddb/currency/EUR HTTP/1.1" 200 357098 1.2474 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.109 Safari/537.36" /var/www/oddb.org.rack/log/2017/07/11/google_crawler_log:80.218.53.88 - - [11/Jul/2017 09:38:39] "GET http://generika.cc/fr/generika/search/zone/drugs/search_query/Metaneuron+N/search_type/st_oddb/currency/EUR HTTP/1.1" 200 356858 1.5794 "Mozilla/5.0 (compatible; google-crawler"
Checked again on 1 PM for crawler entries.
grep -i googlebot log/2017/07/11/user_log | tail 66.249.64.23 - - [11/Jul/2017 13:03:15] "GET http://mobile.oddb.org/de/mobile/fachinfo/reg/52866/chapter/pregnancy HTTP/1.1" 200 16767 0.0836 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.64.17 - - [11/Jul/2017 13:03:17] "GET http://swissmedinfo.oddb.org/de/swissmedinfo/fachinfo/reg/61213/chapter/effects HTTP/1.1" 200 98993 0.4558 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.64.17 - - [11/Jul/2017 13:03:18] "GET http://swissmedinfo.oddb.org/de/swissmedinfo/print/reg/49786/seq/02/patinfo/ HTTP/1.1" 200 19629 0.0384 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.70.19 - - [11/Jul/2017 13:03:18] "GET http://oekk.oddb.org/fr/oekk/print/reg/61245/seq/01/patinfo/ HTTP/1.1" 200 11100 0.0336 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.64.19 - - [11/Jul/2017 13:04:49] "GET http://swissmedinfo.oddb.org/de/swissmedinfo/fachinfo/reg/54732/chapter/interactions HTTP/1.1" 200 18055 0.0835 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.64.17 - - [11/Jul/2017 13:04:49] "GET http://swissmedinfo.oddb.org/de/swissmedinfo/fachinfo/reg/45305/chapter/restrictions HTTP/1.1" 200 35165 0.0758 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.64.17 - - [11/Jul/2017 13:04:50] "GET http://swissmedinfo.oddb.org/de/swissmedinfo/fachinfo/reg/57899 HTTP/1.1" 200 50225 0.0746 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.70.23 - - [11/Jul/2017 13:04:51] "GET http://i.oddb.org/fr/mobile/print/reg/27353/seq/02/patinfo/ HTTP/1.1" 200 11112 0.0331 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.64.19 - - [11/Jul/2017 13:05:40] "GET http://swissmedinfo.oddb.org/de/swissmedinfo/fachinfo/reg/61213/chapter/registration_owner HTTP/1.1" 200 12524 0.0288 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 66.249.64.17 - - [11/Jul/2017 13:05:43] "GET http://swissmedinfo.oddb.org/de/swissmedinfo/fachinfo/reg/27279 HTTP/1.1" 200 40268 0.3080 "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Should they filtered by entries in the apache conf oder by an entry in robots.txt?
We want to move all crontab entries to use the oddb.org version with rack. But the /var/www/oddb.org/data/ contain 18G and I suspect that scripts and constants inside the ruby code reference this path. Therefore I think we should now move the old
Then we should also adapt the services and rename oddb.org -> oddb.org.pre_rack.
Also the doc and log directories have very different sizes
du -sxh /var/www/oddb.org*/doc/ 128M /var/www/oddb.org.rack/doc/ 857M /var/www/oddb.org/doc/ du -sxh /var/www/oddb.org*/log 501M /var/www/oddb.org.rack/log 23G /var/www/oddb.org/log du -sxh /var/www/oddb.org*/data 172M /var/www/oddb.org.rack/data 18G /var/www/oddb.org/data
We must port migel to use Ruby 2.4.0. Currently it uses Ruby 1.9.3 and running bundle exec rake spec shows no problem.
After calling under ruby 2.4.0 bundle install the spec task fails with require 'sbsm/drbserver', as the DrbServer of SBSM is no longer available.
Pushed commit Update to Ruby 2.4.0 and SBSM 1.5.2. Installing it locally and adapting /service/migeld/run to use the line exec sudo -u apache /usr/local/bin/bundle-240 exec ruby-240 bin/migeld. This does not work and chokes with
sudo -u apache bundle-240 exec ruby-240 bin/migeld
The PGconn, PGresult, and PGError constants are deprecated, and will be
removed as of version 1.0.
You should use PG::Connection, PG::Result, and PG::Error instead, respectively.
Called from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/ydbd-pg-0.5.3/lib/dbd/Pg.rb:157:in `new'
FEHLER: Relation »object« existiert bereits
FEHLER: Relation »prefetchable_index« existiert bereits
FEHLER: Relation »extent_index« existiert bereits
FEHLER: Relation »object_connection« existiert bereits
FEHLER: Relation »target_id_index« existiert bereits
FEHLER: Relation »collection« existiert bereits
/usr/local/ruby-2.4.0/lib/ruby/2.4.0/delegate.rb:387:in `__getobj__': not delegated (ArgumentError)
from /usr/local/ruby-2.4.0/lib/ruby/2.4.0/delegate.rb:340:in `block in delegating_block'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/dbi-0.4.5/lib/dbi/handles/statement.rb:214:in `fetch'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/dbi-0.4.5/lib/dbi/handles/statement.rb:240:in `each'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/dbi-0.4.5/lib/dbi/handles/database.rb:130:in `block in select_all'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/dbi-0.4.5/lib/dbi/handles/database.rb:88:in `execute'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/dbi-0.4.5/lib/dbi/handles/database.rb:128:in `select_all'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/ydbd-pg-0.5.3/lib/dbd/pg/database.rb:173:in `columns'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/dbi-0.4.5/lib/dbi/handles/database.rb:161:in `columns'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/odba-1.1.2/lib/odba/connection_pool.rb:39:in `block in method_missing'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/odba-1.1.2/lib/odba/connection_pool.rb:29:in `next_connection'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/odba-1.1.2/lib/odba/connection_pool.rb:38:in `method_missing'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/odba-1.1.2/lib/odba/storage.rb:526:in `setup'
from /var/www/migel/vendor/bundle/ruby/2.4.0/gems/odba-1.1.2/lib/odba/cache.rb:474:in `setup'
from /var/www/migel/lib/migel/persistence/odba.rb:21:in `<module:Migel>'
from /var/www/migel/lib/migel/persistence/odba.rb:15:in `<top (required)>'
from /var/www/migel/lib/migel/persistence.rb:9:in `require'
from /var/www/migel/lib/migel/persistence.rb:9:in `<module:Migel>'
from /var/www/migel/lib/migel/persistence.rb:8:in `<top (required)>'
from bin/migeld:10:in `require'
from bin/migeld:10:in `<main>'
Must fix bin/admin and bin/migeld.
With commit Fixed used versions for Postgres DBI bin/migeld starts without problems and I can search migel products.
With commit Ported bin/admin to rack based SBSM I am able to use the bin/admin
sudo -u apache bundle-240 exec bin/admin migel> migelids.size -> 571 migel> migelids.first -> ["01.01.01.00.1", #<ODBA::Stub:47359814078720#24 @odba_class=Migel::Model::Migelid @odba_container=47359816062800#32286>] migel> migelids.last -> undefined method `last' for #<Hash:0x005625a206aea0> migel> migelids.keys.last -> 99.20.01.00.1 migel> migelids.keys.first -> 01.01.01.00.1 migel>
Installing it on thinpower with these steps
cd /var/www/migel git pull https://github.com/ngiger/migel.git /usr/local/bin/bundle-240 install --path=vendor vi /service/migel/run # adapt it svc -h /service/migel tail -f /service/migeld/log/main/current # to check that it starts
Visited pages like http://ch.oddb.org/de/gcc/migel_search/migel_product/100101012 to verify that oddb.org still works.
We still have yusd and currencyd that run with ruby 1.8. Can we update them to ruby 2.4.0 and what about de.oddb.org?
Pushed two commits for currency, released new version 1.0.2 and changed /service/currency/run to call exec sudo -u apache /usr/local/ruby-2.4.0/bin/currencyd after calling /usr/local/bin/gem-240 install ycurrency. Checked it on http://ch.oddb.org/de/gcc/search/zone/drugs/search_query/Aspirin/search_type/st_oddb#best_result (by changing the currency and looking for changes prices).
The source code contains a file install.rb which seems to be generated automatically, but I have no idea how it was really generated. Also when running it I get errors like
bundle exec ruby install.rb config
install.rb:77:in `<class:ConfigTable>': uninitialized constant Config (NameError)
Did you mean? RbConfig
from install.rb:75:in `<main>'
Should I try to generate a real gem for it? Zeno finds it okay. Also must replace rmail by mail and get rid of the smtp_tls. Implemented the needed changes.
When trying to run the unit tests, I see that I must replace the old Mock.new -> flexmock.
Pushed commit Fix unit tests by clearing session on SessionStore.new to fix the unit tests on sbsm. Release SBSM 1.5.3.
Pushed Update to use SBSM 1.5.3 to make limit queries work. Pulled the changes on thinpower, called bundle install and restarted the services. Now the query limit works.
It did not need any additional work. On oddb-ci2 using in /service/yus/run I use the line exec sudo -u apache /usr/local/bin/ruby-240 /usr/local/src/yus/bin/yusd to start the yus daemon after calling sudo /usr/local/bin/gem-240 install yus. Login/out works as seen by
tail -f /service/yus/log/main/current @400000005964b9872ecc85ac FEHLER: Relation »extent_index« existiert bereits @400000005964b9872ed0a45c FEHLER: Relation »object_connection« existiert bereits @400000005964b9872ed6b324 FEHLER: Relation »target_id_index« existiert bereits @400000005964b9872ed9b894 FEHLER: Relation »collection« existiert bereits @400000005964b9880b1040a4 I, [2017-07-11T13:41:50.185570 #21183] INFO -- start: starting yus-server on drbssl://localhost:9997 @400000005964b9a0313a9b94 I, [2017-07-11T13:42:14.825850 #21183] INFO -- Yus::Server: Login attempt for info@desitin.ch from oddb.org @400000005964b9a0313c2dec W, [2017-07-11T13:42:14.826007 #21183] WARN -- Yus::Server: Authentication failed for info@desitin.ch @400000005964b9a717c7ca9c I, [2017-07-11T13:42:21.398909 #21183] INFO -- Yus::Server: Login attempt for info@desitin.ch from oddb.org @400000005964b9a717c8bccc I, [2017-07-11T13:42:21.399010 #21183] INFO -- Yus::Server: Authentication succeeded for info@desitin.ch @400000005964bb3c360673b4 I, [2017-07-11T13:49:06.906352 #21183] INFO -- Yus::Server: Logout for #<Yus::EntitySession:0x00557242716880>
Installing it on thinpower, too. Seems to work fine for ch.oddb.org and de.oddb.org
Got the following backtrace
TypeError: no implicit conversion of nil into String /var/www/oddb.org.rack/src/view/paypal/redirect.rb:38:in `escape' /var/www/oddb.org.rack/src/view/paypal/redirect.rb:38:in `http_headers' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/state.rb:241:in `view' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/state.rb:173:in `to_html' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:541:in `to_html' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:280:in `block in process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:209:in `synchronize' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:209:in `process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/app.rb:127:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/static.rb:149:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:86:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread'
Corrected watir paypal test to skip the remaining test only when the paypal login is displayed and not at the beginning of each step.
Also adding a test, where we create a new user login for oddb.org, as the current tests use all the customer-1@ywesee.com. Added this test to watir tests and it produces the same backtrace as the one reported by Zeno.
Pushed commit Fix paying with paypal using a new customer as soon as the new watir tests passed. Found the following entries in /service/yus/log/main/current
400000005964cda50438873c I, [2017-07-11T15:07:39.070764 #9055] INFO -- Yus::Server: Login attempt for tst-1499778441@ywesee.com from oddb.org @400000005964cda5043965e4 W, [2017-07-11T15:07:39.070850 #9055] WARN -- Yus::Server: Authentication failed for tst-1499778441@ywesee.com @400000005964cda504737914 I, [2017-07-11T15:07:39.074652 #9055] INFO -- Yus::AutoSession: create_entity(name=tst-1499778441@ywesee.com, valid_until=, valid_from=2017-07-11 15:07:39 +0200) @400000005964cda50b4d0f0c I, [2017-07-11T15:07:39.189552 #9055] INFO -- Yus::Server: Login attempt for tst-1499778441@ywesee.com from oddb.org @400000005964cda50b4e0524 I, [2017-07-11T15:07:39.189640 #9055] INFO -- Yus::Server: Authentication succeeded for tst-1499778441@ywesee.com
Looks to me as if a new user was correctly generated. Now I must check, that after the payment we can proceed with this login.
Fixed password creation with commit Fixed pw when creating new paypal test user and added a puts.
When restarting on thinpower I did see the following error in the log/2017/07/11/user.log
NoMethodError: undefined method `user_agent' for nil:NilClass /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:398:in `is_crawler?' /var/www/oddb.org.rack/src/util/session.rb:153:in `process_late' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:276:in `block in process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:209:in `synchronize' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/session.rb:209:in `process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.3/lib/sbsm/app.rb:127:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/static.rb:149:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:86:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread'
Pushed commits