20111118-prefetch-odba-update-link-ebook-setup-script-ruby193

<< | Index | >>

1. Fix Fachinformation
2. Fix Ebook link
3. Report ODBA.cache.prefetch suspend
4. Setup script for oddb.org on Ruby 193

Goal/Estimate/Evaluation

• Check prefetch and speed / 70% / 60%
• Setup oddb.org script on Ruby 1.9.3 / 70% / 50%

Milestones

1. ODBA prefetch cache test suspend
2. Setup script for ruby 1.9.3

Commit

• Replaced Fachinfo Parse Error link without pointer (oddb.org)

Fix Fachinformation

Email

• ch.ODDB.org Report - Fachinfo Parse Error Reg: 55278, Language; fr - 11/2011

Problem

• The word file is not recognized by OpenOffice

Procedure

1. convert it into the old MS-Word version (.doc)
2. fix it as follows

before

 Effets indésirables
 Affections du système immunitaire
 Affections gastro-intestinales

after

 Effets indésirables
 Affections du système immunitaire
 Affections gastro-intestinales

Result

• http://oddb.masa.org/fr/gcc/fachinfo/reg/55278

Fix Ebook link

Problem

• The link of Ebook does not work on Windows

Update

• bin/decorators/just_medical_de/fr.rb

      def self.decorate model
        model.chapters.each_with_index do |chap, i|
          if chap.heading =~ /Zulassungsinhaberin/
            if chap.paragraphs.first.to_s =~ /AstraZeneca AG/
              table = Text::Table.new
              #table.rows[0][0] = Text::LinkedParagraph.new('http:/www.astrazeneca.ch', 'AstraZeneca AG')
              table.rows[0][0] = Text::LinkedParagraph.new('http://www.astrazeneca.ch', 'AstraZeneca AG')
              table.rows[0] << Text::Paragraph.new(chap.paragraphs.first.to_s.gsub(/AstraZeneca AG/,''))
              chap.paragraphs[0] = table

Report ODBA.cache.prefetch

Log

init system
init system: 5.13363
setup drb-delegation
reset
reset: 5.25701
ODBA.cache.prefetch start
Thu Nov 17 16:57:53 +0100 2011
ODBA.cache.prefetch done
Thu Nov 17 17:05:02 +0100 2011
system initialized
initialized: 434.011449

• http://oddb.masa.org/resources/downloads/status

 2011-11-18 07:25:35: sessions:    0 - threads:    6 - memory: 1446MB 

Note

• It takes about 10 minutes to cache every object (Registration, Sequence, Package, ActiveAgent)
• and about 1.5 GB memory is required
• I do not feel it is faster to search drugs, but the other searching, for example, Wirkstoff, is faster.
• oddbd becomes unstable, which means Segmentation fault often happens

Test ODBA.cache.prefetch on Ruby 1.9.3

Restore

$ sudo -u postgres dropdb oddb.org; sudo -u postgres createdb -E UTF8 -T template0 oddb.org
$ zcat oddb.org_database.bak.20111104.sql.gz | psql -U  potgres oddb.org

• src/util/oddbapp.rb

    def initialize opts={}
 ...
 puts "ODBA.cache.prefetch start"
 puts Time.now
 ODBA.cache.prefetch
 puts "ODBA.cache.prefetch done"
 puts Time.now
      puts "system initialized"
      puts "initialized: #{Time.now - start}"
    end

Run

• bin/oddbd

$ ruby193 -I ../oddb/lib bin/oddbd

Log

ODBA.cache.prefetch start
2011-11-18 07:48:17 +0100
ODBA.cache.prefetch done
2011-11-18 07:52:31 +0100
system initialized
initialized: 260.53717457

Status

masa@masa ~/ywesee/oddb.org.ruby193 $ head doc/resources/downloads/status 
  2011-11-18 07:53:11: sessions:    0 - threads:    6 - memory:  871MB 

Note

• The memory requirement is half than on Ruby 1.8

Benchmark

Without ODBA.cache.prefetch

• First time

masa@masa ~/work $ /usr/sbin/ab -n 500 -c 100 http://oddb.masa.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking oddb.masa.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:        Apache
Server Hostname:        oddb.masa.org
Server Port:            80

Document Path:          /
Document Length:        15673 bytes

Concurrency Level:      100
Time taken for tests:   115.646 seconds
Complete requests:      500
Failed requests:        21
   (Connect: 0, Receive: 0, Length: 21, Exceptions: 0)
Write errors:           0
Non-2xx responses:      21
Total transferred:      7707058 bytes
HTML transferred:       7507367 bytes
Requests per second:    4.32 [#/sec] (mean)
Time per request:       23129.115 [ms] (mean)
Time per request:       231.291 [ms] (mean, across all concurrent requests)
Transfer rate:          65.08 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.9      0       3
Processing: 11727 22036 4664.3  21087   30505
Waiting:    11727 22036 4664.3  21087   30505
Total:      11727 22036 4664.0  21089   30505
WARNING: The median and mean for the initial connection time are not within a normal deviation
        These results are probably not that reliable.

Percentage of the requests served within a certain time (ms)
  50%  21089
  66%  23430
  75%  26092
  80%  27475
  90%  29201
  95%  29740
  98%  29984
  99%  30212
 100%  30505 (longest request)

• Second time

masa@masa ~/work $ /usr/sbin/ab -n 500 -c 100 http://oddb.masa.org/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking oddb.masa.org (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Finished 500 requests


Server Software:        Apache
Server Hostname:        oddb.masa.org
Server Port:            80

Document Path:          /
Document Length:        15673 bytes

Concurrency Level:      100
Time taken for tests:   198.094 seconds
Complete requests:      500
Failed requests:        9
   (Connect: 0, Receive: 0, Length: 9, Exceptions: 0)
Write errors:           0
Non-2xx responses:      9
Total transferred:      7897882 bytes
HTML transferred:       7695443 bytes
Requests per second:    2.52 [#/sec] (mean)
Time per request:       39618.857 [ms] (mean)
Time per request:       396.189 [ms] (mean, across all concurrent requests)
Transfer rate:          38.93 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0       2
Processing:  2530 38032 8968.6  41863   48241
Waiting:     2530 38032 8968.6  41863   48241
Total:       2533 38032 8968.2  41863   48241

Percentage of the requests served within a certain time (ms)
  50%  41863
  66%  44220
  75%  45040
  80%  45311
  90%  46201
  95%  46899
  98%  47981
  99%  48191
 100%  48241 (longest request)

Status

masa@masa ~/ywesee/oddb.org $ head doc/resources/downloads/status 
  2011-11-18 08:29:35: sessions:  971 - threads:   16 - memory: 1743MB 

• With ODBA.cache.prefetch

suspend

Setup script for oddb.org on Ruby 193

Local gem list

masa@masa ~/ywesee/oddb.org $ ruby193 /usr/bin/gem list

*** LOCAL GEMS ***

activesupport (3.1.0)
akami (1.0.0)
archive-tarsimple (1.1.1)
bigdecimal (1.1.0)
builder (3.0.0)
dbd-pg (0.3.9)
dbi (0.4.5)
deprecated (2.0.1)
facets (1.8.54)
flexmock (0.9.0)
gd2 (1.1.1)
gruff (0.3.6)
gyoku (0.4.4)
hpricot (0.8.4)
htmlgrid (1.0.0)
httpi (0.9.5)
i18n (0.6.0)
io-console (0.3)
json (1.5.4)
mechanize (2.0.1)
minitest (2.5.1)
money (3.7.1)
multi_json (1.0.3)
net-http-digest_auth (1.1.1)
net-http-persistent (1.9)
nokogiri (1.5.0)
nori (1.0.2)
odba (1.0.0)
parseexcel (0.5.2)
paypal (2.0.0)
pg (0.12.0pre258)
rack (1.3.4)
rake (0.9.2.2)
rake-compiler (0.7.9)
rclconf (1.0.0)
rdoc (3.9.4)
rmagick (2.12.2)
rmail (1.0.0)
rockit (0.7.2)
rpdf2txt (0.8.2)
ruby-ole (1.2.11.2)
rubyzip (0.9.4)
savon (0.9.7)
sbsm (1.0.0)
spreadsheet (0.6.5.9)
swissmedic-diff (0.1.3)
tmail (1.2.7.1)
turing (0.0.11)
wasabi (2.0.0)
webrobots (0.0.12)
ydim (1.0.0)
yus (1.0.0)

Change parts

• All source files

# encoding: utf-8

• ruby193rc1/lib/ruby/gems/1.9.1/gems/gd2-1.1.1/lib/gd2.rb

 }.inject({}) { |x, (k, v)| x[k] = LIB[name_for_symbol(k, v)]; x }
 #}.inject({}) { |x, (k, v)| x[k] = LIB[name_for_symbol(k, v), v]; x }

• oddb.org/src/plugin/plugin.rb

 #require 'ftools'
 require 'fileutils'

• rockit/token.rb

 # encoding: ascii-8bit

• rockit/rockit_grammars_parser.rb

 # encoding: ascii-8bit

• comment out charactor-encodings (src/model/text.rb)

 #require 'encoding/character/utf-8'

• ruby193rc1/lib/ruby/gems/1.9.1/gems/rpdf2txt-0.8.2/lib/rpdf2txt/object.rb

 require 'digest/md5'

• ruby193rc1/lib/ruby/gems/1.9.1/gems/rpdf2txt-0.8.2/lib/rpdf2txt/parser.rb

 # encoding: utf-8
 require 'digest/md5'

• ruby193rc1/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/handles/statement.rb

 #fetched_rows.push(row)
 fetched_rows.push(row.to_a)

• ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/18_19_loading_compatibility.rb

 #new!(ajd, of, sg)
 new

• add the Cell old definition in ruby193rc1/lib/ruby/1.9.1/csv.rb

 class CSV

1. The version of the installed library.

  VERSION = "2.4.8".freeze

  # deprecated
  class Cell < String
    def initialize(data = "", is_null = false)
      super(is_null ? "" : data)
    end

    def data
      to_s
    end
  end

• src/state/drug/init.rb

 def init
   super
   @model = OpenStruct.new
   fachinfos = @session.app.sorted_fachinfos
   if newest = fachinfos.first
     revision = newest.revision
     date = Time.local(revision.year, revision.month, revision.day)
     day = 24 * 3600
     range = (date-day)...(date+day)
     @model.fachinfo_news = fachinfos.select { |fi|
        #range.include? fi.revision
        true
     }
   end
   @model.feedbacks = @session.app.sorted_feedbacks[0,5]
 end

• ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/18_19_loading_compatibility.rb

 #new!(ajd, of, sg)
 ajd += 1.to_r/2
 jd(ajd)

Compile htmlgrid.so

• replace STR2CSTR by StringValuePtr in grid.c
• compile

 gems/htmlgrid-1.0.0/ext/htmlgrid $ ruby193 extconf.rb 
 gems/htmlgrid-1.0.0/ext/htmlgrid $ make

• gems/htmlgrid-1.0.0/lib/htmlgrid/grid.rb

 begin
   VERSION = '1.0.0'
   ext_dir = File.expand_path('../../ext/htmlgrid', File.dirname(__FILE__))
   ext_lib = File.join(ext_dir, 'htmlgrid.so')
   require ext_lib
 rescue

Setup script

1. Attach:install_ruby193.sh.20111118.txt
2. Attach:install_gems.sh.20111118.txt
3. Attach:install_git.sh.20111118.txt
4. Attach:add_magic_comment.rb.20111118.txt
5. Attach:patches.sh.20111118.txt

Patches

• Attach:gd2.rb.patch.20111118.txt
• Attach:plugin.rb.patch.20111118.txt
• Attach:grid.c.patch.20111118.txt
• Attach:grid.rb.patch.20111118.txt
• Attach:text.rb.patch.20111118.txt
• Attach:token.rb.patch.20111121.txt
• Attach:rockit_grammars_parser.rb.patch.20111121.txt