Many errors can be found looking for start_thread in the log files of the rack processes. Here the number of occurences of yesterday
tart_thread /var/www/oddb.org.rack/log/2017/07/11/*_log /var/www/oddb.org.rack/log/2017/07/11/crawler_log:343 /var/www/oddb.org.rack/log/2017/07/11/evidentia_log:1 /var/www/oddb.org.rack/log/2017/07/11/generika_log:26 /var/www/oddb.org.rack/log/2017/07/11/google_crawler_log:1 /var/www/oddb.org.rack/log/2017/07/11/just-medical_log:2 /var/www/oddb.org.rack/log/2017/07/11/user_log:202
One of the errors is (found 15 time in user_log)
NoMethodError: undefined method `user_agent' for nil:NilClass /var/www/oddb.org.rack/src/util/session.rb:164:in `is_mobile_app?' /var/www/oddb.org.rack/src/util/session.rb:154:in `process_late' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/session.rb:276:in `block in process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/session.rb:209:in `synchronize' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/session.rb:209:in `process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/app.rb:127:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/static.rb:149:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:86:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread'
Fixed with commit Fix is_mobile_app?. Added unit tests for it
Another one is (found 271 times in crawler_log and 83 in user_log)
113.80.98.78 - - [12/Jul/2017 08:31:32] "HEAD http://ch.oddb.org/fr/gcc/result/ HTTP/1.1" 200 20787 0.0451 "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" [2017-07-12 08:31:32] ERROR Rack::Lint::LintError: Response body was given for HEAD request, but should be empty /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:20:in `assert' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:688:in `verify_content_length' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:716:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/body_proxy.rb:36:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/body_proxy.rb:36:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:708:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:110:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread' 113.80.98.78 - - [12/Jul/2017 08:31:33] "GET http://ch.oddb.org/fr/gcc/result/ HTTP/1.1" 200 20787 0.0168 "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" 113.80.98.78 - - [12/Jul/2017 08:31:35] "HEAD http://ch.oddb.org/fr/gcc/ywesee_contact/ HTTP/1.1" 200 7443 0.0244 "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" [2017-07-12 08:31:35] ERROR Rack::Lint::LintError: Response body was given for HEAD request, but should be empty /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:20:in `assert' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:688:in `verify_content_length' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:716:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/body_proxy.rb:36:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/body_proxy.rb:36:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:708:in `each' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:110:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread'
This one puzzles me, because I do not now when a HEAD request is produced by oddb.org. Looks like mostly MS explorer tries to fetch only the HEAD. The error can be reproduced by calling curl -v --HEAD http://oddb-ci2.dyndns.org/fr/gcc/ywesee_contact/
This should be fixed with the commits
Another on is (found 60 times in crawler_log and 91 in user_log)
Encoding::CompatibilityError: incompatible character encodings: UTF-8 and ASCII-8BIT /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/htmlgrid-1.1.4/lib/htmlgrid/template.rb:79:in `block in html_head' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/cgi/html.rb:55:in `nO_element' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/cgi/html.rb:62:in `nO_element_def' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/htmlgrid-1.1.4/lib/htmlgrid/template.rb:71:in `html_head' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/htmlgrid-1.1.4/lib/htmlgrid/template.rb:110:in `block in template_html' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/cgi/html.rb:55:in `nO_element' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/cgi/html.rb:62:in `nO_element_def' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/cgi/html.rb:423:in `html' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/htmlgrid-1.1.4/lib/htmlgrid/template.rb:109:in `template_html' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/htmlgrid-1.1.4/lib/htmlgrid/template.rb:122:in `to_html' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/state.rb:174:in `to_html' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/session.rb:541:in `to_html' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/session.rb:280:in `block in process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/session.rb:209:in `synchronize' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/session.rb:209:in `process_rack' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/app.rb:127:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/static.rb:149:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:86:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread'
This one can be reproduced by visiting http://ch.oddb.org/fr/gcc/show/reg/54064/seq/10/pack/300
This is caused by a hoover for the index_therapeuticus that contains an ' in Préparations contre l'alcoolisme et le tabagisme'
Fixed with commit Display tooltips correctly which contain an apostrophe. Eg. fr/54064
Found the following problem with fachinfo rss
37.187.56.76 - - [12/Jul/2017 10:40:15] "GET http://hirslanden.oddb.org/de/hirslanden/rss/channel/sl_introduction.rss HTTP/1.1" 500 61058 0.0268 "Mozilla/5.0 (X11; Linux i686; rv:21.0) Gecko/20100101 Firefox/21.0" Errno::ENOENT: No such file or directory @ rb_file_s_size - /var/www/oddb.org.rack/data/rss/de/fachinfo-hirslanden.rss /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.5/lib/sbsm/app.rb:133:in `size' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.5/lib/sbsm/app.rb:133:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/static.rb:149:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:86:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread'
Same error for
Visiting http://generika.cc/de/generika/rss/channel/sl_introduction.rss leads to this problem
Errno::ENOENT: No such file or directory @ rb_file_s_size - /var/www/oddb.org.rack/data/rss/de/sl_introduction-generika.rss /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/app.rb:133:in `size' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/sbsm-1.5.4/lib/sbsm/app.rb:133:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/static.rb:149:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call' /var/www/oddb.org.rack/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:86:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run' /usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread' 85.3.24.2 - - [12/Jul/2017 09:55:17] "GET http://generika.cc/de/generika/rss/channel/sl_introduction.rss HTTP/1.1" 500 62105 0.0582 "Apple-PubSub/65.28"
Here we are missing a rewrite rule, as the file /var/www/oddb.org.rack/data/rss/de/sl_introduction-generika.rss does not exist, but /var/www/oddb.org.rack/data/rss/de/sl_introduction.rss should be returned.
Fixed with commits
When accessing the main page with a user_agent like iphone we should choose the mobile (aka i.ch.oddb.org). This can be tested with curl -v --user-agent "Mozilla/5.0 (iPhone; CPU iPhone OS 10_2_1 like Mac OS X) AppleWebKit/602.4.6 (KHTML, like Gecko) Version/10.0 Mobile/14D27 Safari/602.1" http://ch.oddb.org/
Pushed commit Added watir test for mobile flavor. The error lays in the apache conf where the redirection to port 8012 must come after the checks for iPhone.
The IP is constructed in the method query_limit_explain of src/view/limit.rb. It uses the session.remote_ip, which is calculated in sbsm/lib/session.rb in the method process_rack(rack_request:) as follows
@remote_ip = rack_request.env['HTTP_X_FORWARDED_FOR']
It looks like here we have sometimes two entries.
Pushed commits: