view · edit · sidebar · attach · print · history

20120224-fix-migel-view-atc-overview-oddb_org-drop-crawler-request-sbsm

<< Masa.20120227-check-accessor-class-ActiveAgent-AtcClass-Sequence-fix-ATCOverview-error-oddb_org | Index | Masa.20120223-update-mail_order_price-debug-errors-oddb_org-update-grid_rb-htmlgrid >>

  1. Fix grid.rb htmlgrid
  2. Debug ATC overview in search result oddb.org
  3. Drop clawler request after 50 threads oddb.org
  4. Check data floating oddb.org
Commits
  1. Fix migel result view for grid.rb to work correctly (oddb.org)
  2. Fix ATC Overview in Medikamente search result (oddb.org)
  3. Fix ATC Overview in Medikamente search result (oddb.org)
  4. Drop crawler requests if drb server threads goes over 50 (sbsm)
  5. Refactor lib/sbsm/request.rb. Non cralwer requests (normal requests) skip the check process of /tmp/sbsm_drop_flag file (sbsm)
  6. Skip the threads check process if status file is not found in the check of drb server threads (sbsm)

Fix grid.rb for migel htmlgrid

Migel search result

  • grid.c
  • grid.rb

Commit

Debug ATC overview in search result oddb.org

Refer to http://dev.ywesee.com/wiki.php/Masa/20120223-update-mailOrderPrice-debug-errors-oddbOrg-update-gridRb-htmlgrid#DebugATC

Commit

Drop clawler request after 50 threads oddb.org

Commit

Note

  • Every request refers to /tmp/sbsm_drop_flag file that includes 'true' or 'false'
  • The number of threads on DRb server (oddbd) goes up over 50, or drop down under 50, this file will be updated
  • The process to check the number of threads on DRb server runs once in 5 seconds in average ('Time.now.sec.to_i%5==0' is the condition to start the check process)
  • Under 50 threads, the requests to match crawler pattern sleep 2 seconds
  • Over 50 threads, all the crawler requests are dropped out (not to be sent to oddbd)
  • A normal request (non crawler pattern matched request) is processed as normal

Check data floating oddb.org

Problem

  • Sometimes data floating happens after updating data via a job script

z.B. 

@400000004f4657dc38032a2c error in SBSM::Session#http_headers:
/de/gcc/search/search_query/Emoton+beta+Cimicifuga
@400000004f4657dc380331fc NoMethodError
@400000004f4657dc380331fc undefined method `values' for ",":String
@400000004f4657dc380335e4
/var/www/oddb.org/src/model/sequence.rb:241:in `public_packages'

Note

  • After the rebooting, this sort of error will be fixed
  • This indicates that the odba_id is maybe mixed together by updating data via a job script
  • because a job script uses the different memory space, different odba_id list, from oddb main application
  • Or many threads accesses at the same time may confuses ODBA and it causes the data floating on the cache

Experiment

  • bin/admin 
ch.oddb> ODDB::Package.find_by_pharmacode('223332').sequence.packages['017'] = 'hogehoge'
-> hogehoge

Error 

error in SBSM::Session#process: /de/gcc/search/zone/drugs/search_query/inderal/search_type/st_oddb
NoMethodError
undefined method `public?' for "hogehoge":String
/home/masa/ywesee/oddb.org/src/model/sequence.rb:252:in `block in public_package_count'
/home/masa/ywesee/oddb.org/src/model/sequence.rb:251:in `each'
/home/masa/ywesee/oddb.org/src/model/sequence.rb:251:in `public_package_count'

Note

  • This error is fixed after rebooting oddbd
  • Also ODBA.cache.clear can resolve this error
  • The question is
    • 'when does this data floating on the cache happens?'
    • 'how do I reproduce the data floating?'

Experiment 

@400000004f40f1ab05f43d14 error in SBSM::Session#to_html: /en/oekk/search/search_query/C09B
@400000004f40f1ab05f43d14 NoMethodError
@400000004f40f1ab05f43d14 undefined method `registration' for "utf-8":String
@400000004f40f1ab05f444e4 /var/www/oddb.org/src/model/package.rb:28:in `block (2 levels) in registration_data'
  • bin/admin 
ch.oddb> ODDB::Package.find_by_pharmacode('223332').sequence = 'utf-8'
-> utf-8

Result 

error in SBSM::Session#http_headers: /de/gcc/search/zone/drugs/search_query/inderal/search_type/st_oddb
NoMethodError
undefined method `registration' for "utf-8":String
/home/masa/ywesee/oddb.org/src/model/package.rb:28:in `block (2 levels) in registration_data'

Note

  • I can reproduce the similar NoMethodError as we logged before online
  • This operation is possible, because Package#sequence is :attr_accessor
  • Most of the attributes shouldbe :attr_reader if possible, then it becomes impossbile to update the data on the cache (output error and we can know wherer the bug is)

Solution

  • Check the class of attr_accessor variable when a value is set to the instance variable

Example

  • src/model/package.rb 
    def sequence=(arg, klass=ODDB::Sequence)
      if arg.is_a?(klass)
        @sequence = arg
      else
        arg = if arg.respond_to?(:to_s)
                arg.to_s[0, 10]
              end
        raise TypeError.new("#{arg.to_s} should be #{klass}")
      end
    end
  • bin/admin 
ch.oddb> ODDB::Package.find_by_pharmacode('223332').sequence = 'utf-8'
-> utf-8 should be ODDB::Sequence
ch.oddb> ODDB::Package.find_by_pharmacode('223332').sequence
-> Inderal 10 mg, Tabletten

Note

  • The updating value is prohibited if the class is not appropriate
view · edit · sidebar · attach · print · history
Page last modified on February 24, 2012, at 05:18 PM