Day Three

<< DayTwo | Index | >>

• htmlrid crash: Approach 2 -- stare at the grid.c code and try to spot a bug. The function around grid.c:403 seems to be correct (at least when considering ~2 function call levels below), as long as the inputs are correct.
• htmlgrid crash: Approach 1 -- try to find a request that crashes oddb.

Looked at apache error log for clusters of errors. Usually the errors were grouped in ~30 second windows, so that may mean server segfaulted and got restarted. After trying ~10 such request no luck crashing oddbd. Possibly this means the bug is non-deterministic. The other possibility is that the actual request was not logged in apache error_log/access_log.

In any case, a good idea is to log all request coming into oddbd, along with timestamps.

• protip -- use firefox Export Cookies plugin to export cookies for use with 'wget --load-cookies'
• crash of oddb version 2a90ba22634b1724c9e74853e2097679ed82ab0f (c7a2924264f5362e6098d449fa385c65c485e6a3 was ok) :

        /usr/local/lib/ruby/gems/1.9.1/gems/odba-1.0.8/lib/odba/marshal.rb:15:in `load': undefined class/module ODDB::Migel::Product (ArgumentError)

=> needed new database.

• protip - piping stderr into a file (examples)

 # pipe all output into a file
 user@localhost ~/currency $ /usr/local/bin/ruby bin/currencyd &> | tee stdout_with_stderr.log
 # tee all output into a file (as usual, tee output to console may lag, due to (not) flushing inside the program)
 user@localhost ~/currency $ /usr/local/bin/ruby bin/currencyd 2>&1 | tee stdout_with_stderr.log
 # pipe stdout (unix file descriptor 1) and stderr (unix file descriptor 2) into separate files
 user@localhost ~/currency $ /usr/local/bin/ruby bin/currencyd 1>stdout.log 2>stderr.log

• Benchmarking mod_ruby compiled against ruby 1.8 and 1.9 has similar results, ruby 1.8 seems just a little bit faster. In massively concurrent situations this builds up to more failed requests with ruby 1.9
• note - both mod_ruby-1.3 and mod_ruby-git, when configured with /usr/local/ruby1.9, link statically to libruby

=> reinstall ruby1.9 (configure --enable-shared), and recompile mod_ruby

 $ ldd mod_ruby_1.9.so 
 linux-gate.so.1 =>  (0xffffe000)
 libruby.so.1.9 => /usr/local/lib/libruby.so.1.9 (0xb74f0000)
 libpthread.so.0 => /lib/libpthread.so.0 (0xb74c6000)
 librt.so.1 => /lib/librt.so.1 (0xb74bd000)
 libdl.so.2 => /lib/libdl.so.2 (0xb74b9000)
 libcrypt.so.1 => /lib/libcrypt.so.1 (0xb7487000)
 libm.so.6 => /lib/libm.so.6 (0xb7461000)
 libc.so.6 => /lib/libc.so.6 (0xb7303000)
 /lib/ld-linux.so.2 (0xb777d000)

• going anywhere from the oddb index pages results in error_log :

 [Fri Jan 13 10:39:22 2012] [error] mod_ruby: /usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in  require': /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.0.3/lib/sbsm/flavored_uri_parser.rb:9: invalid multibyte char (US-ASCII) (SyntaxError)

=> add to the file

 # -*- coding: ascii-8bit -*-

• any request now shows (the source of) /var/www/oddb.org/doc/index.rbx

=> 'qfile mod_ruby' shows that we are missing a configuration file. Copy the standard gentoo's 21_mod_ruby.conf (https://gist.github.com/1605296) to /etc/apache2/modules.d, then LoadModule in /etc/apache2/vhosts.d/01_oddb_vhosts.conf is no longer needed

• security error in sbsm (apache error_log)

 [Fri Jan 13 09:23:27 2012] [error] mod_ruby: /usr/local/lib/ruby/gems/1.9.1/gems/sbsm-1.0.3/lib/sbsm/trans_handler.rb:21:in `expand_path': Insecure operation - expand_path (SecurityError)

=> replace File.dirname(__FILE__) with File.dirname(__FILE__).untaint on lines 21 and 23, add '.untaint' to 'request.server.document_root' on line 29

• checking apache2 with mod_ruby/1.9

 $ sudo /etc/init.d/apache2 start
 * apache2 has detected an error in your setup:
 Syntax error on line 17 of /etc/apache2/vhosts.d/01_oddb_vhost.conf:
 Invalid command 'RubyAddPath', perhaps misspelled or defined by a module not included in the server configuration
 * ERROR: apache2 failed to start

=> add on top of /etc/apache2/vhosts.d/01_oddb_vhost.conf :

 LoadModule ruby_module /usr/lib/apache2/modules/mod_ruby.so

=> apache now starts, but after loading oddb main page the following error appears in /var/log/apache2/error_log :

 [Fri Jan 13 08:55:03 2012] [error] mod_ruby: failed to require sbsm/trans_handler
 [Fri Jan 13 08:55:03 2012] [error] mod_ruby: error in ruby
 [Fri Jan 13 08:55:03 2012] [error] mod_ruby: /usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': /home/user/.local/rubygems/lib/rockit/token.rb:139: invalid multibyte char (UTF-8) (SyntaxError)
 [Fri Jan 13 08:55:03 2012] [error] mod_ruby: /home/user/.local/rubygems/lib/rockit/token.rb:139: syntax error, unexpected '~', expecting ')'
 [Fri Jan 13 08:55:03 2012] [error] mod_ruby:     super("EOF", "\xa4~~\xa4\xa4~^^~" + rand(1e10).inspect)

=> as per http://dev.ywesee.com/wiki.php/Masa/20110118-update-rpdf2txt, add magic comment:

 # -*- coding: ascii-8bit -*-

to files:

 /home/user/.local/rubygems/lib/rockit/rockit_grammars_parser.rb
 /home/user/.local/rubygems/lib/rockit/token.rb

• compiling mod_ruby against ruby1.9

 $ emerge -C mod_ruby
 $ git clone https://github.com/shugo/mod_ruby.git
 $ cd mod_ruby
 $ ./configure.rb # same as /usr/local/bin/ruby configure.rb
 $ make

=> gives the error:

 /usr/include/apache2/ap_config.h:25:17: fatal error: apr.h: No such file or directory

=> add '-I/usr/include/apr-1' to APACHE_INCLUDES line in Makefile, and run 'make' again, then 'sudo make install'