20120319-print-view-error-create-second-process-for-crawler

<< | Index | >>

summary

• checked print-view error.
• created separate druby process for search engine crawler
  • tried on example project.
  • applied update to oddb.org

commit

oddb.org

• Added second oddbd process for crawler
• Fixed login for separate process

sbsm

• Re-allow windows user to first process

shoe

• Attach:shoe-crawler-process-20120319.patch.txt
• Added second process for crawler.
• Updated access_test as test from access_log
• Updated to create status file by process
• Fixed session problem

index

• Error on Print View
• Separate process for search engine
  • try with rewrite_rule
  • test step
• Add second process to oddb.org
• Setup process with deamontools
• Ajax compare search

Error on Print View

Problem

@400000004f64b06e055a4ed4 error in SBSM::Session#http_headers:
/en/mobile/print/fachinfo/57656
@400000004f64b06e055a5e74 NoMethodError
@400000004f64b06e055a5e74 undefined method `name' for "":String
@400000004f64b06e055a625c
/var/www/oddb.org/src/view/printtemplate.rb:53:in `name'
@400000004f64b06e055a625c

URL

http://oddb.yasuhiro.org/en/mobile/print/fachinfo/57656

But Now, There are No error.
It seems that this error is related in data-floating.

suspend

Separate process for search engine

First, tested on example project shoe.

try with rewrite_rule

tested to switch flavor by user-agent with RewriteRule of mod_rewrite

  RewriteEngine on
  RewriteCond %{REQUEST_URI} !\.(html|pdf|csv|xls|js|css|jpg|png)$ 
  RewriteCond %{HTTP_USER_AGENT}  ^Mozilla
  RewriteRule ^/$ /de/seo/home [PT,L]
  RewriteRule ^/([defren]+)/([A-z_\-]+)/(.*)$ /$1/seo/$3 [PT,L]

refes

• http://httpd.apache.org/docs/current/mod/mod_rewrite.html
• http://httpd.apache.org/docs/current/rewrite/flags.html#flag_pt

It works. But I found is_crawler? method in sbsm. then I thought next way.

try with using sbsm request

in /path/to/sbsm/request.rb

...
   def is_crawler?
      warn @cgi.user_agent
      crawler_pattern = /archiver|slurp|bot|crawler|jeeves|spider|\.{6}|windows/i
      !!crawler_pattern.match(@cgi.user_agent)
    end
...

created separate process, then added request handling in index.rbx

Attach:shoe-crawler-process-20120319.patch.txt (this update has session problem)

see this commit

yasu      8741  1.3  1.1 111764 93664 pts/3    Sl+  09:05   0:47 Oddb (OddbApp)
yasu     13069 56.0  0.9  94568 77564 pts/13   Sl+  10:05   0:18 Oddb (OddbApp:Crawler)

tested with

• https://addons.mozilla.org/de/firefox/addon/user-agent-switcher/

test step

in shoe project

• 1. patch adding separate process for crawler
  • Attach:Shoe-Added-second-process-for-crawler.patch-20120319.txt
• 2. patch updating access_test
  • Attach:Shoe-Updated-access_test-as-test-from-access_log.patch-20120319.txt
• 4. run server application
  • bin/shoed
  • bin/shoed crawler
• 3. create data
  • run test/data_loader.rb
• 4. create simple access_log
  • run test/access_log_generator.rb
• 5. test access
  • run test/batch.sh
  • or run test/access_test.rb
• 6. check status
  • run test/status.sh
  • see test/log/status

  2012-03-19 13:46:36: sessions:   63 - threads:   81  - memory:  125MB 
  2012-03-19 13:46:35: sessions:   10 - threads:   96  - memory:  141MB 
  2012-03-19 13:46:31: sessions:   63 - threads:   64  - memory:  108MB 
  2012-03-19 13:46:30: sessions:   10 - threads:   74  - memory:  119MB 

Add second process to oddb.org

With same method, created second process for crawler.
I had to update index.rbx. because SBSM::Request makes session.
(If I make request object with no DrbServer URI, Session are gone.)

index.rbx

(Not Works)

  request = SBSM::Request.new(nil)
  if (request.is_crawler?)
    server_uri = SHOE::SERVER_URI_FOR_CRAWLER
  else
    server_uri = SHOE::SERVER_URI
  end 
  SBSM::Request.new(server_uri).process

(It works)

  request = SBSM::Request.new(ODDB::SERVER_URI)
  if request.is_crawler?
    request = SBSM::Request.new(ODDB::SERVER_URI_FOR_CRAWLER)
  end 
  request.process

Now, oddb.arg has 2 status output file.

• http://oddb.yasuhiro.org/resources/downloads/status
• http://oddb.yasuhiro.org/resources/downloads/status_crawler

Setup process with deamontools

In production server, We use daemontools.
I created new run script in to /var/www/oddb.org/svc/oddb_crawler

• create
  • svc/oddb_crawler
  • svc/oddb_crawler/log/main
  • svc/oddb_crawler/log/run (mtlog)
  • svc/oddb_crawler/run (service)
  • /servic/SERVICE_NAME (symbolic link)
• reboot
  • svc -h /secvice/SERVICE_NAME
• check
  • svstat /service/SERVICE_NAME

NOTE:

• don't update run-script directly. (create run.new and replace with mv and use -t option)

Ajax compare search

Problem

http://oddb.yasuhiro.org/de/gcc/compare_search

only iPhone, tap highlight dosn't works.

tried this.
But only a few times works...

-webkit-tap-highlight-color: rgba(0,0,0,0);

continue

refs

• http://www.yuiblog.com/blog/2010/10/01/quick-tip-customizing-the-mobile-safari-tap-highlight-color/