20111014-AZsearch-encoding-ruby193-oddb_org

<< | Index | >>

1. Check date oddb.org on Ruby 1.9.3
2. Debug htmlgrid.so encoding error
3. Set all the String to UTF-8

Goal/Estimate/Evaluation

• Arzneimittel AZ search / 50% / 50%

Milestones

1. Arzneimittel Search oddb.ch
   • htmlgrid.so encoding error
2. force_encoding every string data

Check date oddb.org on Ruby 1.9.3

Run

 masa@masa ~/ywesee/oddb.org $ ruby193 -I ../oddb/lib bin/oddbd

Access

• http://oddb.masa.org/de/gcc/show/reg/31706/seq/01/pack/017

Result

• Looks good
• The date is the same as the online server

Debug htmlgrid.so encoding error

Change log

• http://dev.ywesee.com/wiki.php/Choddb/Ruby193RC1#AZSearch

Note

• Now the Arzneimittel AZ search works by force_encoding the specific variables in Sequence class but
• Fundamentally all the data has different encodings in the database

Set all the String to UTF-8

Task

• Improve migrate_utf8 method

Experiment

• src/util/oddbapp.rb

    def migrate_to_utf8
      #iconv = ::Iconv.new 'UTF-8//TRANSLIT//IGNORE', 'ISO-8859-1'
      iconv = ::Iconv.new 'UTF-8//TRANSLIT//IGNORE', 'ASCII'

    def _migrate_to_utf8 queue, table, iconv, opts={}
      ...
      obj.odba_store unless obj.odba_unsaved?
      rescue
    end

    def _migrate_obj_to_utf8 obj, queue, table, iconv, opts={}
      ...
      rescue
    end

Run

ruby186 bin/admin

bin/admin
ch.oddb> migrate_to_utf8

Note

• Some marshal data was not possible to be loaded
• All errors are skipped

Result

/usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:324: [BUG] Segmentation fault
ruby 1.8.6 (2009-06-08) [x86_64-linux]

Abgebrochen

Run on Ruby 1.9.3

ruby193 -I ../oddb/lib bin/oddbd

bin/admin
ch.oddb> migrate_to_utf8