20111012-recover-order-failure-bbmb

<< Masa.20111013-date-ruby193-encoding-error-htmlgrid_so-oddb_org-missing-drug-search-bbmb_ch | 2011 | Masa.20111011-table-contents-kindle-ebook-htmlgrid_so-ruby193-oddb_org-change-top-address-bbmb_ch >>

1. Kindle ebook with kindlegen v 1.2
2. Recovery order failure bbmb.ch
3. drug search oddb.org on Ruby 1.9.3

Goal/Estimate/Evaluation

• missing polling xmlconv bbmb / 100% / 100%
• drug search oddb.org on Ruby 1.9.3 / 70% / 50%

Milestones

1. bbmb top
2. recover order faiulre

Kindle ebook with kindlegen v 1.2

Email

• kindlestrip - stripping the datasource from a kindlegen 1.2 file

Note

• Success to make a book size less

Recovery order failure bbmb.ch

Refer to http://dev.ywesee.com/wiki.php/Gag/Xmlconvchangelog

drug search oddb.org on Ruby 1.9.3

Problem

• Drug search does not work
• Any results are not shown

Note

• It is sure that there is data

ch.oddb> package('223332').name
-> Inderal 10 mg, Tabletten

• The problem is in the search process

Experiment

• src/state/global.rb

      def _search_drugs_state(query, stype)
      ...
        else
          result = _search_drugs(query, stype)
      p result.package_count

Run

 masa@masa ~/ywesee/oddb.org $ ruby193 -I ../oddb/lib bin/oddbd 

Access

• http://oddb.masa.org/de/gcc/search/zone/drugs/search_query/inderal/search_type/st_oddb#best_result

Result

 0 

Note

• This should be normally '15'

Next

• src/util/oddbapp.rb

    if(atcs.empty?)
      atcs = search_by_sequence(key, result)
    p atcs.length
    p atcs[0].class
    p atcs[0].package_count
      result.search_type = :sequence
    end

Result

 1
 ODDB::AtcClass
 0

Note

• search_by_sequence looks working but the atcs[0].package_count should be '15'
• search_by_sequence method is defined as follows

  def search_by_sequence(key, result=nil)
    ODBA.cache.retrieve_from_index('sequence_index_atc', key.dup, result)
  end

• ODBA.cache.retrieve_from_index works incorrectly

Experiment

(ruby18 bin/oddb)
ch.oddb> search_by_sequence('inderal')[0].package_count
-> 15

(ruby193 -I ../oddb/lib bin/oddbd)
ch.oddb> search_by_sequence('inderal')[0].package_count
-> 0

Note

• rebuild_indices method maybe does not work
• ('rebuild_indices' creates index tables for ODBA.cache.retrieve_from_index method)

Experiment (rebuild index tables for the cache search)

ch.oddb> rebuild_indices

Note

• Many errors

Focus on 'sequence_index_atc' table

ch.oddb> rebuild_indices 'sequence_index_atc'

Log

dropping: sequence_index_atc
creating: sequence_index_atc
filling: sequence_index_atc
@atc_classes.values
source.size: 7369
Encoding::CompatibilityError
incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
/home/masa/ywesee/oddb.org/src/model/sequence.rb:266:in `split'
/home/masa/ywesee/oddb.org/src/model/sequence.rb:266:in `search_terms'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/stub.rb:112:in `method_missing'
(eval):3:in `block in proc_resolve_search_term'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:146:in `call'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:146:in `search_term'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:149:in `search_terms'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:278:in `search_terms'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:70:in `block (2 levels) in fill'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:69:in `each'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:69:in `block in fill'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:66:in `each'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/index.rb:66:in `fill'
/home/masa/bin/ruby193rc1/lib/ruby/gems/1.9.1/gems/odba-1.0.0/lib/odba/cache.rb:330:in `fill_index'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1253:in `block in rebuild_indices'
/home/masa/bin/ruby193rc1/lib/ruby/1.9.1/psych/deprecated.rb:30:in `each'
/home/masa/bin/ruby193rc1/lib/ruby/1.9.1/psych/deprecated.rb:30:in `load_documents'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1230:in `rebuild_indices'
/home/masa/bin/ruby193rc1/lib/ruby/1.9.1/delegate.rb:72:in `method_missing'
(eval):1:in `block (2 levels) in _admin'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1472:in `instance_eval'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1472:in `block (2 levels) in _admin'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `call'
/home/masa/ywesee/oddb.org/src/util/failsafe.rb:9:in `failsafe'
/home/masa/ywesee/oddb.org/src/util/oddbapp.rb:1471:in `block in _admin'
finished in 0.0072754474166666664 min

Experiment

• src/model/sequence.rb

    # encoding: utf-8

    def search_terms
      #str = self.name
      str = self.name.force_encoding('utf-8')
      ODDB.search_terms(str.split(/\s+/u).push(str))
    end

Run

masa@masa ~/ywesee/oddb.org $ ruby193 -I ../oddb/lib bin/oddbd

masa@masa ~/ywesee/oddb.org $ ruby193 -I ../oddb/lib bin/admin 
ch.oddb> rebuild_indices 'sequence_index_atc'
-> 

Log

dropping: sequence_index_atc
creating: sequence_index_atc
filling: sequence_index_atc
@atc_classes.values
source.size: 7369
finished in 1.4783700731166667 min
dropping: sequence_index
creating: sequence_index
filling: sequence_index
self.sequences
source.size: 15271
finished in 1.4111290253666666 min
all Indices Created in total: 0.04815952567555556 h

Check

ch.oddb> search_by_sequence('inderal')[0].package_count
-> 0

Access

• http://oddb.masa.org/de/gcc/search/zone/drugs/search_query/inderal/search_type/st_oddb#best_result

Result

• failed

Note

• 2 possibilites
  1. the index table, sequence_index_atc, is created incorrectly
  2. the index table, sequence_index_atc, is created correctly but ODBA.cache.retrieve_from_index method failed in searching

Look at the table data directly

masa@masa ~/ywesee/oddb.org $ psql -U postgres oddb.org
oddb.org=# \o sequence_index_atc.dat
oddb.org=# select * from sequence_index_atc;
oddb.org=# \q

Result

• The index table looks fine

Experiment

ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences').length
-> 9
ch.oddb> search_by_sequence('inderal')[0].package_count
-> 0
ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[0].name
-> Inderal, Injektionslösung

Note

• Some sequences are picked up by the retrieve_from_index method

Experiment

ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[0].public_packages
-> []
ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[0].packages.values.length
-> 1
ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[0].packages.values[0]
-> #<ODDB::Package:0x000000049a1970>
ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[0].packages.values[0].name
-> Inderal, Injektionslösung
ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[0].public?
-> false
ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[0].packages.values[0].public?
-> true

The differrences

• Ruby 1.8.6

 ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].name
 -> Inderal 10 mg, Tabletten
 ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].public?
 -> true
 ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].public_packages.length
 -> 2

• Ruby 1.9.3

 ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].name
 -> Inderal 10 mg, Tabletten
 ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].public?
 -> false
 ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].public_packages.length
 -> 0

Question

• Why does the Sequence#public? becomes 'false' (Ruby 1.9.3)?
• src/model/sequence.rb

    def public?
      !@export_flag && @registration.public? && active?
    end

Check

ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].registration.iksnr
-> 31706
ch.oddb> search_by_sequence('inderal')[0].instance_variable_get('@sequences')[1].seqnr
-> 01
ch.oddb> registrations['31706'].sequences['01'].public?
-> false
ch.oddb> registrations['31706'].sequences['01'].active?
-> false
ch.oddb> registrations['31706'].sequences['01'].instance_variable_get('@export_flag')
-> false
ch.oddb> registrations['31706'].public?
-> false

Note

• The registration.public? and active? are false
• These should be true
• src/model/sequence.rb

    def active?
      (!@inactive_date || (@inactive_date > @@two_years_ago)) \
        && @registration && @registration.active? && !violates_patent?
    end

Check

ch.oddb> registrations['31706'].sequences['01'].instance_variable_get('@inactive_date')
-> 
ch.oddb> registrations['31706'].sequences['01'].instance_variable_get('@inactive_date').class
-> NilClass
ch.oddb> ODDB::Sequence.class_eval('@@two_years_ago')
-> 2009-10-12
ch.oddb> registrations['31706'].active?
-> false
ch.oddb> registrations['31706'].sequences['01'].violates_patent?
-> false

Note

• The registration.public? should be true
• @inactive_date should not be nil
• This may be related to ODBA::Date._load function