view · edit · sidebar · attach · print · history

20120319-print-view-error-create-second-process-for-crawler

<< | Index | >>


summary

  • checked print-view error.
  • created separate druby process for search engine crawler
    • tried on example project.
    • applied update to oddb.org

commit

oddb.org
sbsm
shoe

index


Error on Print View

Problem
@400000004f64b06e055a4ed4 error in SBSM::Session#http_headers:
/en/mobile/print/fachinfo/57656
@400000004f64b06e055a5e74 NoMethodError
@400000004f64b06e055a5e74 undefined method `name' for "":String
@400000004f64b06e055a625c
/var/www/oddb.org/src/view/printtemplate.rb:53:in `name'
@400000004f64b06e055a625c
URL

http://oddb.yasuhiro.org/en/mobile/print/fachinfo/57656

But Now, There are No error.
It seems that this error is related in data-floating.

suspend


Separate process for search engine

First, tested on example project shoe.

try with rewrite_rule

tested to switch flavor by user-agent with RewriteRule of mod_rewrite

  RewriteEngine on
  RewriteCond %{REQUEST_URI} !\.(html|pdf|csv|xls|js|css|jpg|png)$ 
  RewriteCond %{HTTP_USER_AGENT}  ^Mozilla
  RewriteRule ^/$ /de/seo/home [PT,L]
  RewriteRule ^/([defren]+)/([A-z_\-]+)/(.*)$ /$1/seo/$3 [PT,L]

refes

It works. But I found is_crawler? method in sbsm. then I thought next way.

try with using sbsm request

in /path/to/sbsm/request.rb

...
   def is_crawler?
      warn @cgi.user_agent
      crawler_pattern = /archiver|slurp|bot|crawler|jeeves|spider|\.{6}|windows/i
      !!crawler_pattern.match(@cgi.user_agent)
    end
...

created separate process, then added request handling in index.rbx

Attach:shoe-crawler-process-20120319.patch.txt (this update has session problem)

see this commit

yasu      8741  1.3  1.1 111764 93664 pts/3    Sl+  09:05   0:47 Oddb (OddbApp)
yasu     13069 56.0  0.9  94568 77564 pts/13   Sl+  10:05   0:18 Oddb (OddbApp:Crawler)

tested with

test step

in shoe project

  2012-03-19 13:46:36: sessions:   63 - threads:   81  - memory:  125MB 
  2012-03-19 13:46:35: sessions:   10 - threads:   96  - memory:  141MB 
  2012-03-19 13:46:31: sessions:   63 - threads:   64  - memory:  108MB 
  2012-03-19 13:46:30: sessions:   10 - threads:   74  - memory:  119MB 

Add second process to oddb.org

With same method, created second process for crawler.
I had to update index.rbx. because SBSM::Request makes session.
(If I make request object with no DrbServer URI, Session are gone.)

index.rbx

(Not Works)

  request = SBSM::Request.new(nil)
  if (request.is_crawler?)
    server_uri = SHOE::SERVER_URI_FOR_CRAWLER
  else
    server_uri = SHOE::SERVER_URI
  end 
  SBSM::Request.new(server_uri).process

(It works)

  request = SBSM::Request.new(ODDB::SERVER_URI)
  if request.is_crawler?
    request = SBSM::Request.new(ODDB::SERVER_URI_FOR_CRAWLER)
  end 
  request.process

Now, oddb.arg has 2 status output file.

Setup process with deamontools

In production server, We use daemontools.
I created new run script in to /var/www/oddb.org/svc/oddb_crawler

  • create
    • svc/oddb_crawler
    • svc/oddb_crawler/log/main
    • svc/oddb_crawler/log/run (mtlog)
    • svc/oddb_crawler/run (service)
    • /servic/SERVICE_NAME (symbolic link)
  • reboot
    • svc -h /secvice/SERVICE_NAME
  • check
    • svstat /service/SERVICE_NAME

NOTE:

  • don't update run-script directly. (create run.new and replace with mv and use -t option)

Ajax compare search

Problem

http://oddb.yasuhiro.org/de/gcc/compare_search

only iPhone, tap highlight dosn't works.

tried this.
But only a few times works...

-webkit-tap-highlight-color: rgba(0,0,0,0);

continue

refs

view · edit · sidebar · attach · print · history
Page last modified on March 28, 2012, at 06:25 PM