20111017-migrate_to_utf8-ruby193-oddb_org

<< | Index | >>

1. Fix migrate_to_utf8 bin/admin command
2. Debug Hash iteration error

Goal/Estimate/Evaluation

• migerate_to_utf bin/admin oddb.org / 70% / 50%

Milestones

1. migerate ruby 1.9.3 oddb.org
   1. debug Hash iteration bug in ODBA

Fix migrate_to_utf8 bin/admin command

Current status

• Run on Ruby 1.8.6 -> Segmentation fault
• Run on Ruby 1.9.3 -> Many encoding errors

Note

• Marshaled String on Ruby 1.8 becomes 'ASCII-8BIT' when it is unmarshaled on Ruby 1.9.3.

Run on Ruby 1.8.6

• ruby1.8 bin/oddbd
• bin/admin

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> migrate_to_utf8
-> /usr/lib64/ruby/site_ruby/1.8/odba/marshal.rb:15:in `load': undefined class/module ODDB::IncompleteRegistration
failsafe rescued ArgumentError < StandardError
/usr/lib64/ruby/site_ruby/1.8/odba/marshal.rb:15:in `load': undefined class/module ODDB::IncompleteRegistration
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1471:in `_admin'
/usr/lib64/ruby/site_ruby/1.8/odba/marshal.rb:15:in `load'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:246:in `fetch_collection'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:244:in `each'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:244:in `fetch_collection'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:641:in `restore'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:336:in `fetch_or_restore'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:330:in `call'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:330:in `fetch_or_do'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:335:in `fetch_or_restore'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:649:in `restore_object'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:610:in `load_object'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:232:in `fetch'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:330:in `call'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:330:in `fetch_or_do'
/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:231:in `fetch'
/usr/lib64/ruby/site_ruby/1.8/odba/stub.rb:51:in `odba_instance'
/usr/lib64/ruby/site_ruby/1.8/odba/stub.rb:151:in `respond_to?'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1813:in `_migrate_obj_to_utf8'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1811:in `each'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1811:in `_migrate_obj_to_utf8'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1807:in `_migrate_to_utf8'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1793:in `migrate_to_utf8'
(eval):1:in `_admin'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `call'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `failsafe'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1470:in `_admin'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1468:in `initialize'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1468:in `new'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1468:in `_admin'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `__send__'
/usr/lib64/ruby/1.8/drb/drb.rb:1555:in `perform_without_block'
/usr/lib64/ruby/1.8/drb/drb.rb:1515:in `perform'
/usr/lib64/ruby/1.8/drb/drb.rb:1589:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1585:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1581:in `main_loop'
/usr/lib64/ruby/1.8/drb/drb.rb:1430:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `start'
/usr/lib64/ruby/1.8/drb/drb.rb:1427:in `run'
/usr/lib64/ruby/1.8/drb/drb.rb:1347:in `initialize'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `new'
/usr/lib64/ruby/1.8/drb/drb.rb:1627:in `start_service'
bin/oddbd:38

Note

• There is a mismatch between the saved marshal data and the class structure in oddb.org
  • ODDB::IncompatibleRegistration is deleted in this commit:
  • ODDB::RootUser is deleted in this commit: http://scm.ywesee.com/?p=oddb.org/.git;a=commit;h=7c681c2db82be396761c5826e9ed3d3b4f6f5250
• Run on Ruby 1.9.3 makes #<RuntimeError: can't add a new key into hash during iteration> error
• This should be fixed first, otherwise it does not run even on Ruby 1.8
  • This is a bug in ODBA

Post an article to ruby-list ML (Japanese)

• http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/48476
  • http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/48477 (reply by Nobuoka-san)
  • http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-list/48478 (reply by Niku-san)

Force_encoding String object when unmarshaling (by Nobuoka-san)

  data = Marshal.load( ARGF, Proc.new{|obj|
    if obj.is_a? String
      obj.force_encoding( "UTF-8" )
    end
    obj
  })

Note

• This works but in oddb.org the following error comes

Potential Memory-Leak: stub for ODDB::Hospital#692011 was saved without container
stub for NilClass: was saved with receiver in collection of Hash:689832
repair: remove [689832, 040849221237363031303032303331343632063a064554, 136]
repair: insert [689832, 040849221237363031303032303331343632063a064554, 136]
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/dbi-0.4.5/lib/dbi/typeutil.rb:50: stack level too deep (SystemStackError)

• http://www.google.com/search?q=dbi%2Ftypeutil.rb%3A50%3A%20stack%20level%20too%20deep%20%28SystemStackError%29
• http://stackoverflow.com/questions/5694005/stack-level-too-deep-error-in-rubys-recursive-call

Debug Hash iteration error

Run

• ruby193 -I ../oddb/lib bin/oddbd

#<RuntimeError: can't add a new key into hash during iteration>
["/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:320:in `store'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:320:in `block (3 levels) in fetch_or_restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `call'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `fetch_or_do'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:315:in `block (2 levels) in fetch_or_restore'",
 "<internal:prelude>:10:in `synchronize'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:314:in `block in fetch_or_restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `call'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `fetch_or_do'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:312:in `fetch_or_restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:62:in `block in bulk_restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:59:in `each'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:59:in `bulk_restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:53:in `bulk_fetch'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:255:in `fetch_collection'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:592:in `restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:313:in `block in fetch_or_restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `call'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `fetch_or_do'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:312:in `fetch_or_restore'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:600:in `restore_object'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:565:in `load_object'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:221:in `block in fetch'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `call'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:308:in `fetch_or_do'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:220:in `fetch'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/stub.rb:49:in `odba_receiver'",
 "/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/stub.rb:122:in `respond_to?'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1827:in `block in _migrate_obj_to_utf8'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1819:in `each'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1819:in `_migrate_obj_to_utf8'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1810:in `_migrate_to_utf8'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1796:in `migrate_to_utf8'",
 "(eval):1:in `block (2 levels) in _admin'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1474:in `instance_eval'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1474:in `block (2 levels) in _admin'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/failsafe.rb:9:in `call'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/failsafe.rb:9:in `failsafe'",
 "/home/masa/ywesee/oddb.org.ruby193/src/util/oddbapp.rb:1473:in `block in _admin'"]

Experiment

• odba-1.0.0/lib/odba/cache.rb

    def fetch(odba_id, odba_caller=nil)
p odba_id
      fetch_or_do(odba_id, odba_caller) { 
        load_object(odba_id, odba_caller)
      }
    end

Run

• ruby193 -I ../oddb/lib bin/oddbd
• bin/admin

ch.oddb> migrate_to_utf8

Log

17
#<RuntimeError: can't add a new key into hash during iteration>
17
17
[17
"/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:322:in `store'"17
,17
...

Note

• odba_id 17 is a Hash of ODDB::AtcClass

ch.oddb> ODBA.cache.fetch(17)
-> Hash
ch.oddb> ODBA.cache.fetch(17).values[0].class
-> ODDB::AtcClass
ch.oddb> ODBA.cache.fetch(17).values.length
-> 7369

• It succeeds in fetching 17 on bin/admin
• It looks failed on migrate_to_utf8 script

Experiment

• odba-1.0.0/lib/odba/cache.rb

    def fetch_or_restore(odba_id, dump, odba_caller) # :nodoc:
      ...
      #hash.store(odba_obj.odba_id, cache_entry)
      hash.store(odba_obj.odba_id, cache_entry) unless hash.has_key?(odba_obj.odba_id)

Run

• ruby193 -I ../oddb/lib bin/oddbd
• bin/admin

ch.oddb> migrate_to_utf8

Result

• failed