view · edit · sidebar · attach · print · history

20111121-importer-ruby193-slowsearch-bottleneck-oddb_org

<< Masa.20111122-slowsearch-bottleneck-oddb_org | 2011 | Masa.20111118-prefetch-odba-update-link-ebook-setup-script-ruby193 >>

  1. Check Importer on Ruby 1.9.3
  2. Check ODDB::Log class
  3. Check update_swissmedic suspend
  4. Find out why the search takes long time
Goal/Estimate/Evaluation
  • Importer on Ruby 1.9.3 oddb.org / 50% / 10%
Milestones
  • Check each importer on Ruby 1.9.3 suspend
  • Slow search reason

Check Importer on Ruby 1.9.3

Check update_swissmedic

Run 

masa@masa ~/ywesee/oddb.org.ruby193 $ ruby193rc1 -I ../oddb/lib bin/oddbd

bin/admin 

ch.oddb> Updater.new(self).update_swissmedic

Result 

TypeError: can't convert String into Integer when updating index 'sequence_generic_type_index' with a ODDB::Sequence
["/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `[]'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `block in current_ids'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `collect'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `current_ids'"]
[...]

TypeError: can't convert String into Integer when updating index 'sequence_generic_type_index' with a ODDB::Sequence
["/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `[]'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `block in current_ids'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `collect'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `current_ids'"]
[...]

TypeError: can't convert String into Integer when updating index 'sequence_generic_type_index' with a ODDB::Sequence
["/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `[]'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `block in current_ids'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `collect'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `current_ids'"]
[...]

TypeError: can't convert String into Integer when updating index 'sequence_generic_type_index' with a ODDB::Sequence
["/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `[]'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `block in current_ids'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `collect'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `current_ids'"]
[...]

TypeError: can't convert String into Integer when updating index 'sequence_generic_type_index' with a ODDB::Sequence
["/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `[]'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `block in current_ids'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `collect'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `current_ids'"]
[...]

ERROR:  could not access status of transaction 906726
DETAIL:  Could not open file "pg_clog/0000": No such file or directory.

Oddb (OddbApp): symbol lookup error: /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/tmail-1.2.7.1/ext/tmailscanner/tmail/tmailscanner.so: undefined symbol: rb_get_kcode

Check spreadsheet testcases

masa@masa ~/work/spreadsheet/test $ ruby193 suite.rb 
/usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
Run options: 

# Running tests:

...................................................

Finished tests in 0.738766s, 69.0341 tests/s, 1261.5641 assertions/s.

51 tests, 932 assertions, 0 failures, 0 errors, 0 skips

Check swissmedic-diff testcases

masa@masa ~/work/swissmedic-diff/test $ ruby193 test_swissmedic-diff.rb 
/usr/local/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': iconv will be deprecated in the future, use String#encode instead.
Run options: 

# Running tests:

......

Finished tests in 0.302911s, 19.8078 tests/s, 128.7507 assertions/s.

6 tests, 39 assertions, 0 failures, 0 errors, 0 skips

Note

  • 'iconv' library should be replaced

Check ODDB::Log class

Experiment

  • bin/admin 
ch.oddb> ODDB::Log.new(Date.today).notify('test')
-> connection closed

Log 

Oddb (OddbApp): symbol lookup error: /home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/tmail-1.2.7.1/ext/tmailscanner/tmail/tmailscanner.so: undefined symbol: rb_get_kcode

Next

  • Replace tmail to mail gem library in ODDB::Log class

Patch

Test 

ch.oddb> ODDB::Log.new(Date.today).notify('test')
ch.oddb> ODDB::Log.new(Date.today).notify_attachment(File.read('/home/masa/work/test.xls'), {:mime_type => 'application/octet-stream', :filename => 'test.xls', :subject => 'subject test',})

Reference

Check update_swissmedic

Run 

ch.oddb> Updater.new(self).update_swissmedic

Log 

TypeError: can't convert String into Integer when updating index 'sequence_generic_type_index' with a ODDB::Sequence
["/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `[]'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:304:in `block in current_ids'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `collect'", "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:302:in `current_ids'"]
[...]
...

Note

  • Segmentation fault happens many times

Email 

Plugin: ODDB::SwissmedicPlugin
Error: RuntimeError
Message: can't add a new key into hash during iteration
Backtrace:
/home/masa/ywesee/oddb.org.ruby193/src/model/activeagent.rb:79:in `[]='
/home/masa/ywesee/oddb.org.ruby193/src/model/activeagent.rb:79:in `block in adjust_types'
/home/masa/ywesee/oddb.org.ruby193/src/model/activeagent.rb:67:in `each'
/home/masa/ywesee/oddb.org.ruby193/src/model/activeagent.rb:67:in `adjust_types'
/home/masa/ywesee/oddb.org.ruby193/src/util/persistence.rb:59:in `diff'
/home/masa/ywesee/oddb.org.ruby193/src/util/persistence.rb:287:in `issue_update'
/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:128:in `block in update'
/home/masa/ywesee/oddb.org.ruby193/src/util/failsafe.rb:10:in `call'
/home/masa/ywesee/oddb.org.ruby193/src/util/failsafe.rb:10:in `failsafe'
/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:127:in `update'
/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1464:in `update'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:473:in `update_active_agent'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:522:in `block (2 levels) in update_compositions'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:521:in `each'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:521:in `block in update_compositions'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:513:in `each'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:513:in `each_with_index'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:513:in `update_compositions'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:725:in `block in update_registrations'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:721:in `each'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:721:in `update_registrations'
/home/masa/ywesee/oddb.org.ruby193/src/plugin/swissmedic.rb:44:in `update'
/home/masa/ywesee/oddb.org.ruby193/src/util/updater.rb:347:in `block in update_swissmedic'
/home/masa/ywesee/oddb.org.ruby193/src/util/updater.rb:477:in `call'
/home/masa/ywesee/oddb.org.ruby193/src/util/updater.rb:477:in `wrap_update'
/home/masa/ywesee/oddb.org.ruby193/src/util/updater.rb:345:in `update_swissmedic'
(eval):1:in `block (2 levels) in _admin'
/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1483:in `instance_eval'
/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1483:in `block (2 levels) in _admin'
/home/masa/ywesee/oddb.org.ruby193/src/util/failsafe.rb:10:in `call'
/home/masa/ywesee/oddb.org.ruby193/src/util/failsafe.rb:10:in `failsafe'
/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1482:in `block in _admin'

Updates

  • src/model/activeagent.rb 
    def adjust_types(values, app=nil)
      values = values.dup
      #values.each { |key, value|
      values.dup.each { |key, value|

Check hash iterations 

masa@masa ~/ywesee/oddb.org.ruby193 $ grep -r "each { |key, value|" src/
src/model/activeagent.rb:                       values.dup.each { |key, value| 
src/model/composition.rb:      values.each { |key, value|
src/model/doctor.rb:      values.each { |key, value|
src/model/galenicform.rb:                       values.each { |key, value|
src/model/migel/product.rb:                             hash.each { |key, value|
src/model/package.rb:                   values.each { |key, value|
src/model/part.rb:      values.each { |key, value|
src/model/registration.rb:                      hash.each { |key, value|
src/model/sequence.rb:                  values.each { |key, value|
src/model/slentry.rb:                   values.each { |key, value|
src/model/substance.rb:                 values.each { |key, value|
src/model/substance.rb:                 other.descriptions.dup.each { |key, value|
src/plugin/flockhart.rb:                                @cytochromes.each { |key, value|
src/plugin/flockhart.rb:                                writer.extract_data.each { |key, value|
src/plugin/hayes.rb:                            TABLES.each { |key, value|
src/plugin/interaction.rb:                              @update_reports.each { |key, value|
src/plugin/interaction.rb:                              @hayes.each { |key, value|
src/plugin/interaction.rb:                              @flockhart.each { |key, value|
src/plugin/medwin.rb:                   @errors.each { |key, value|
src/plugin/medwin.rb:                   @errors.each { |key, value|
src/state/feedbacks.rb:                 hash.each { |key, value|
src/util/persistence.rb:                        adjust_types(values, app).each { |key, value|
src/util/persistence.rb:                        values.each { |key, value|

Find out why the search takes long time

Experiment

  • src/view/drub/result.rb 
 def init
...
p "before super"
p Time.now
    super
p "end super"
p Time.now

Log 

"before super"
Mon Nov 21 16:07:46 +0100 2011
"end super"
Mon Nov 21 16:11:30 +0100 2011
view · edit · sidebar · attach · print · history
Page last modified on April 04, 2012, at 08:59 AM