view · edit · sidebar · attach · print · history

20130605-jobsDaily

<< | Index | >>

Summary

  • Make brand2csv run over 10'000 entries
  • Try to make jobs-daily work on funtoo
  • Try to upgrade yus to work with ruby 1.9

Commits

Index

Make brand2csv run over 10'000 entriess

  • Retry did not work as expected. Got the following error 
"01:41:10: Opening log/detail_P-484026.html using https://www.swissreg.ch/srclient/faces/jsp/trademark/sr300.jsp?language=de&section=tm&id=P-484026"
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `initialize'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `open'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `block in connect'
/home/niklaus/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/timeout.rb:54:in `timeout'
/home/niklaus/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/timeout.rb:99:in `timeout'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:29:in `connect'
/home/niklaus/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:755:in `do_start'
/home/niklaus/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/1.9.1/net/http.rb:750:in `start'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:628:in `start'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:570:in `connection_for'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:926:in `request'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/mechanize-2.7.0/lib/mechanize/http/agent.rb:263:in `fetch'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/mechanize-2.7.0/lib/mechanize.rb:431:in `get'
/home/niklaus/.rvm/gems/ruby-1.9.3-p392/gems/mechanize-2.7.0/lib/mechanize.rb:441:in `get_file'
/opt/src/brand2csv/lib/brand2csv.rb:342:in `fetchDetails'
/opt/src/brand2csv/lib/brand2csv.rb:517:in `block in fetchresult'
/opt/src/brand2csv/lib/brand2csv.rb:513:in `each'
/opt/src/brand2csv/lib/brand2csv.rb:513:in `fetchresult'
/opt/src/brand2csv/lib/brand2csv.rb:541:in `run'
bin/brand2csv:45:in `<main>'

Fixing it required creating a new agent and not reusing the same again.

Pushed https://github.com/zdavatz/brand2csv/commit/fd72bb77a2f5652484961119f01c55dc7a77e512 Fixed (re-)opening of session

Swissreg seems to artificially slow down responses as soon as 100 hits have been sent by the same IP.

After about 3 hours and 1416 fetched details I got another error 

/home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:90:in `initialize': getaddrinfo: Name or service not known (SocketError)
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:90:in `open'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:90:in `connect'
        from /home/niklaus/.rvm/rubies/ruby-1.8.7-p371/lib/ruby/1.8/timeout.rb:53:in `timeout'
        from /home/niklaus/.rvm/rubies/ruby-1.8.7-p371/lib/ruby/1.8/timeout.rb:101:in `timeout'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/net-http-persistent-2.8/lib/net/http/persistent/ssl_reuse.rb:90:in `connect'
        from /home/niklaus/.rvm/rubies/ruby-1.8.7-p371/lib/ruby/1.8/net/http.rb:553:in `do_start'
        from /home/niklaus/.rvm/rubies/ruby-1.8.7-p371/lib/ruby/1.8/net/http.rb:548:in `start'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:628:in `start'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:570:in `connection_for'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/net-http-persistent-2.8/lib/net/http/persistent.rb:926:in `request'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/mechanize-2.7.1/lib/mechanize/http/agent.rb:257:in `fetch'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/mechanize-2.7.1/lib/mechanize.rb:431:in `get'
        from /home/niklaus/.rvm/gems/ruby-1.8.7-p371/gems/mechanize-2.7.1/lib/mechanize.rb:441:in `get_file'
        from /opt/src/brand2csv/lib/brand2csv.rb:138:in `init_swissreg'
        from /opt/src/brand2csv/lib/brand2csv.rb:524:in `fetchresult'
        from /opt/src/brand2csv/lib/brand2csv.rb:514:in `each'
        from /opt/src/brand2csv/lib/brand2csv.rb:514:in `fetchresult'
        from /opt/src/brand2csv/lib/brand2csv.rb:542:in `run'
        from bin/brand2csv:45

Added delay of 60 seconds after exception 'getaddrinfo: Name or service not known', as the whole import takes many hours.

Make the jobs/import_daily work on vagrant funtoo

  • Changed in jobs/import_daily the shebang from #!/usr/local/bin/ruby193 to #!/usr/bin/env ruby1.9.

Logging standard output and error. Daily-Import started normally (at least for my eyes), see below 

/usr/lib64/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
racc/parser.rb:33: warning: already initialized constant Racc_Runtime_Version
racc/parser.rb:34: warning: already initialized constant Racc_Runtime_Revision
racc/parser.rb:36: warning: already initialized constant Racc_Runtime_Core_Version_R
racc/parser.rb:37: warning: already initialized constant Racc_Runtime_Core_Revision_R
racc/parser.rb:41: warning: already initialized constant Racc_Runtime_Core_Revision_C
racc/parser.rb:49: warning: already initialized constant Racc_Main_Parsing_Routine
racc/parser.rb:50: warning: already initialized constant Racc_YY_Parse_Method
racc/parser.rb:51: warning: already initialized constant Racc_Runtime_Core_Version
racc/parser.rb:52: warning: already initialized constant Racc_Runtime_Core_Revision
racc/parser.rb:53: warning: already initialized constant Racc_Runtime_Type
[DEPRECATED] By requiring 'spreadsheet/excel' you are loading a Compatibility
             layer which provides a drop-in replacement for Spreadsheet::Excel
             versions <= 0.3.5.1. This code will be removed in Spreadsheet
             version 1.0.0

/var/www/oddb.org/src/view/changelog.rb:27: warning: already initialized constant SORT_DEFAULT
/var/www/oddb.org/src/view/drugs/resultlist.rb:24: warning: already initialized constant BACKGROUND_SUFFIX
could not find htmlgrid.so, falling back to pure-ruby class
ERROR:  relation "object" already exists
<..>
-> "220 edge04.upcmail.net edge ESMTP server ready\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-edge04.upcmail.net hello [84.73.49.252], pleased to meet you\r\n"
-> "250-HELP\r\n"
-> "250-AUTH LOGIN PLAIN\r\n"
-> "250-SIZE 31457280\r\n"
-> "250-ENHANCEDSTATUSCODES\r\n"
-> "250-8BITMIME\r\n"
-> "250-STARTTLS\r\n"
-> "250 OK\r\n"
<- "STARTTLS\r\n"
-> "220 2.0.0 Ready to start TLS\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-edge04.upcmail.net hello [84.73.49.252], pleased to meet you\r\n"
-> "250-HELP\r\n"
-> "250-AUTH LOGIN PLAIN\r\n"
-> "250-SIZE 31457280\r\n"
-> "250-ENHANCEDSTATUSCODES\r\n"
-> "250-8BITMIME\r\n"
-> "250 OK\r\n"
<- "AUTH PLAIN AG5pa2xhdXMuZ2lnZXJAaGlzcGVlZC5jaABuZzEyMzQ=\r\n"
-> "235 2.7.0 ... authentication succeeded\r\n"
<- "MAIL FROM:<niklaus.giger@hispeed.ch>\r\n"
-> "250 2.1.0 <niklaus.giger@hispeed.ch> sender ok\r\n"
<- "RCPT TO:<>\r\n"
-> "550 5.5.0 <> invalid address\r\n"
<- "QUIT\r\n"
-> "221 2.0.0 edge04.upcmail.net edge closing connection\r\n"
-> "220 edge03.upcmail.net edge ESMTP server ready\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-edge03.upcmail.net hello [84.73.49.252], pleased to meet you\r\n"
-> "250-HELP\r\n"
-> "250-AUTH LOGIN PLAIN\r\n"
-> "250-SIZE 31457280\r\n"
-> "250-ENHANCEDSTATUSCODES\r\n"
-> "250-8BITMIME\r\n"
-> "250-STARTTLS\r\n"
-> "250 OK\r\n"
<- "STARTTLS\r\n"
-> "220 2.0.0 Ready to start TLS\r\n"
<- "EHLO ywesee.com\r\n"
-> "250-edge03.upcmail.net hello [84.73.49.252], pleased to meet you\r\n"
-> "250-HELP\r\n"
-> "250-AUTH LOGIN PLAIN\r\n"
-> "250-SIZE 31457280\r\n"
-> "250-ENHANCEDSTATUSCODES\r\n"
-> "250-8BITMIME\r\n"
  • Problem: Mail in /var/www/oddb.org/test/testenvironment1.rb was not set correctly. Changed the lines like this. 
host  = 'smtp.hispeed.ch'
mail = 'niklaus.giger@hispeed.ch'

Restarted import. This did not work, as it complained that the previous job was still running, even if it was not alive. 

loading testenvironment

DEVELOPER_HOST = smtp.hispeed.ch
DEVELOPER_MAIL = niklaus.giger@hispeed.ch

disabling UPDATER
Process 5437 is running

But error-mail reached me. Which is good.

After rm /var/www/oddb.org/log/job.pid, restarting the job worked. import_daily finished without an error and without sending me an email after 39 minutes. What is going wrong?

Try to upgrade yus to work with ruby 1.9

  • Installed yus via git clone git://scm.ywesee.com/yus
  • Added Gemfile to install replace gem18 install postgres needle flexmock
  • ruby18 -rubygems test/suite.rb has neither errors nor failures
  • Same for ruby19 report 10 errors, but all report NoMethodError: undefined method `critical=' for Thread:Class 
ruby18 -rubygems test/suite.rb
NoMethodError: undefined method `critical=' for Thread:Class
    /home/vagrant/.gem/ruby/1.9.1/gems/needle-1.3.0/lib/needle/thread.rb:42:in `lock'
    <internal:prelude>:8:in `synchronize'
    /home/vagrant/.gem/ruby/1.9.1/gems/needle-1.3.0/lib/needle/service-point.rb:117:in `instance'
    /home/vagrant/.gem/ruby/1.9.1/gems/needle-1.3.0/lib/needle/container.rb:308:in `get'
    /home/vagrant/.gem/ruby/1.9.1/gems/needle-1.3.0/lib/needle/registry.rb:124:in `bootstrap'
    /home/vagrant/.gem/ruby/1.9.1/gems/needle-1.3.0/lib/needle/registry.rb:106:in `initialize'
    /opt/src/yus/lib/yus/server.rb:14:in `new'
    /opt/src/yus/lib/yus/server.rb:14:in `initialize'
    /opt/src/yus/test/test_server.rb:28:in `new'
    /opt/src/yus/test/test_server.rb:28:in `setup'

Conclusion: We have to update/replace the 'needle' gem. The version 1.3.0 dates from December 24, 2005. 

ruby-1.9.3-p392 test/ALL-TESTS.rb 
/opt/src/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `initialize' may cause serious problems
/opt/src/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `object_id' may cause serious problems
/opt/src/needle-1.3.0/lib/needle/definition-context.rb:36: warning: undefining `__send__' may cause serious problems
/home/niklaus/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- models/model_test (LoadError)
        from /home/niklaus/.rvm/rubies/ruby-1.9.3-p392/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require'
        from models/tc_threaded.rb:21:in `<top (required)>'
        from test/ALL-TESTS.rb:21:in `load'
        from test/ALL-TESTS.rb:21:in `block in <main>'
        from test/ALL-TESTS.rb:21:in `each'
        from test/ALL-TESTS.rb:21:in `<main>'
from http://ruby-doc.org/core-2.0/Thread.html

exclusive { block } => obj click to toggle source Wraps a block in Thread.critical, restoring the original value upon exit from the critical section, and returns the value of the block.

http://ruby-doc.org/core-1.8.7/Thread.html had a Thread.critical= condition, but not yet an exclusive method.

But we should answer first the question, whether we want to base the next yus version on a gem which is no longer supported. If yes, we should fork it and make it run under Ruby 1.9.

The other alternative would be do redesign yus to avoid dependency injection and use another method, see discussion http://kresimirbojcic.com/2011/11/19/dependency-injection-in-ruby.html and the Ruby alternative in https://gist.github.com/1379464

view · edit · sidebar · attach · print · history
Page last modified on June 05, 2013, at 04:31 PM