20100921-trace-import-limitation-process

<< Masa.20100922-trace-limitation-data-flow-in-bsv_xml | 2010 | Masa.20100920-trace-limitationText-object >>

1. ODBA Error Report
2. grep limitation text
3. Read bsv_xml.rb
4. Study REXML Parser library
5. Trace bsv_xml.rb

Goal

• Trace import limitation_text process / 80%

Milestones

1. grep limitation_text 8:45
2. Look at
   • src/plugin/bsv_xml.rb
   • src/plugin/analysis.rb cancel
   • etc.

Summary

• http://masay.ch/2010/09/21/trace-import-limitation-process/

Commits

ToDo Tomorrow

1. Dimdi-Import-Error.

Keep in Mind

• emerge --sync
• 20100907 Ticket 242, Comment 9 - notification Mail
• 20100908 rpdf2txt GPLv3
• database name of ch.oddb.org => oddb.org
• block sample http://q.hatena.ne.jp/1217573703 (japanese)
• check ydpm priorities

Attached Files

ODBA Error Report

Mail

• ch.ODDB.org Report - Error Export: fachinfo.yaml - 09/2010

Error: ODBA::OdbaError
Message: Unknown odba_id 23772297
Backtrace:
(druby://localhost:10005) /usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:598:in `restore_object'
(druby://localhost:10005) /usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:565:in `load_object'
(druby://localhost:10005) /usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:221:in `fetch'
(druby://localhost:10005) /usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:308:in `call'
(druby://localhost:10005) /usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:308:in `fetch_or_do'
(druby://localhost:10005) /usr/lib64/ruby/site_ruby/1.8/odba/cache.rb:220:in `fetch'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:234:in `export_yaml'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:233:in `each'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:233:in `export_yaml'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:255:in `call'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:255:in `safe_export'
(druby://localhost:10005) /usr/lib64/ruby/1.8/tempfile.rb:172:in `open'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:254:in `safe_export'
(druby://localhost:10005) /var/www/oddb.org/ext/export/src/odba_exporter.rb:232:in `export_yaml'
/var/www/oddb.org/src/plugin/yaml.rb:16:in `export_array'
/var/www/oddb.org/src/plugin/yaml.rb:25:in `export_fachinfos'
/var/www/oddb.org/src/util/exporter.rb:202:in `export_yaml'
/var/www/oddb.org/src/util/exporter.rb:289:in `call'
/var/www/oddb.org/src/util/exporter.rb:289:in `safe_export'
/var/www/oddb.org/src/util/exporter.rb:201:in `export_yaml'
/var/www/oddb.org/src/util/schedule.rb:14:in `call'
/var/www/oddb.org/src/util/schedule.rb:14:in `run_on_weekday'
/var/www/oddb.org/src/util/exporter.rb:200:in `export_yaml'
/var/www/oddb.org/src/util/exporter.rb:50:in `run'
/var/www/oddb.org/jobs/export_daily:13
/var/www/oddb.org/src/util/job.rb:17:in `call'
/var/www/oddb.org/src/util/job.rb:17:in `run'
/var/www/oddb.org/jobs/export_daily:12

grep limitation text

List

masa@masa ~/ywesee/oddb.org $ grep -r limitation_text src
src/custom/lookandfeelbase.rb:                          :limitation_text                                        =>      'Limitationstext',
src/custom/lookandfeelbase.rb:                          :limitation_texts                                       =>      'Limitationstexte',
src/custom/lookandfeelbase.rb:                          :limitation_text_title0         =>      'Limitationen zu ',
src/custom/lookandfeelbase.rb:                          :limitation_text_title1         =>      ':',
src/custom/lookandfeelbase.rb:                          :th_limitation_text                             =>      'Limitationstext',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts0        =>      'Limitationstexte "',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts1        =>      '", ',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts2        =>      ' von Total ',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts3        =>      '.',
src/custom/lookandfeelbase.rb:                          :limitation_text                                        =>      'Texte de limitation',
src/custom/lookandfeelbase.rb:                          :limitation_texts                                       =>      'Textes de limitation',
src/custom/lookandfeelbase.rb:                          :limitation_text_title0         =>      'Limitations à ',
src/custom/lookandfeelbase.rb:                          :limitation_text_title1         =>      ':',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts0        =>      'Textes de limitation "',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts1        =>      '", ',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts2        =>      ' de ',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts3        =>      ' en total.',
src/custom/lookandfeelbase.rb:                          :limitation_text                                        =>      'Limitation text',
src/custom/lookandfeelbase.rb:                          :limitation_texts                                       =>      'Limitation texts',
src/custom/lookandfeelbase.rb:                          :limitation_text_title0         =>      'Limitation to ',
src/custom/lookandfeelbase.rb:                          :limitation_text_title1         =>      ':',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts0        =>      'Limitationtexts "',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts1        =>      '", ',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts2        =>      ' of ',
src/custom/lookandfeelbase.rb:                          :title_limitation_texts3        =>      ' in total.',
src/custom/lookandfeelbase.rb:        [0,0] =>  :limitation_text,
src/custom/lookandfeelbase.rb:                          [0,0]           =>      :limitation_text,
src/custom/lookandfeelwrapper.rb:                               [0,0]           =>      :limitation_text,
src/custom/lookandfeelwrapper.rb:                       :limitation_texts,
src/custom/lookandfeelwrapper.rb:        [0,0] =>       :limitation_text,
src/custom/lookandfeelwrapper.rb:                               [0,0]           =>      :limitation_text,
src/custom/lookandfeelwrapper.rb:                               [0,0]           =>      :limitation_text,
src/custom/lookandfeelwrapper.rb:        [7,0]   =>  :limitation_text,
src/custom/lookandfeelwrapper.rb:      :limitation_texts,
src/custom/lookandfeelwrapper.rb:        [0,0]          =>      :limitation_text,
src/custom/lookandfeelwrapper.rb:                               [0,0]           =>      :limitation_text,
src/custom/lookandfeelwrapper.rb:        [0,0]          =>      :limitation_text,
src/model/analysis/position.rb:                 attr_reader :limitation_text, :footnote, :list_title, 
src/model/analysis/position.rb:                 def create_limitation_text
src/model/analysis/position.rb:                         @limitation_text = LimitationText.new
src/model/analysis/position.rb:                 def delete_limitation_text
src/model/analysis/position.rb:                         if(lt = @limitation_text)
src/model/analysis/position.rb:                                 @limitation_text = nil
src/model/analysis/position.rb:                                 @limitation_text, self].compact.collect { |doc|
src/model/index_therapeuticus.rb:    attr_reader :code, :comment, :limitation_text 
src/model/index_therapeuticus.rb:    def create_limitation_text
src/model/index_therapeuticus.rb:      @limitation_text = LimitationText.new
src/model/index_therapeuticus.rb:    def delete_limitation_text
src/model/index_therapeuticus.rb:      if(lt = @limitation_text)
src/model/index_therapeuticus.rb:        @limitation_text = nil
src/model/migel/group.rb:                       attr_accessor :limitation_text, :group
src/model/migel/group.rb:                               @limitation_text.odba_delete unless(@limitation_text.nil?)
src/model/migel/group.rb:                       def create_limitation_text
src/model/migel/group.rb:                               @limitation_text = LimitationText.new
src/model/migel/group.rb:                       def delete_limitation_text
src/model/migel/group.rb:                               if(lt = @limitation_text)
src/model/migel/group.rb:                                       @limitation_text = nil
src/model/migel/product.rb:                             :qty, :unit, :limitation_text
src/model/migel/product.rb:                             @limitation_text.odba_delete unless(@limitation_text.nil?)
src/model/migel/product.rb:                     def create_limitation_text
src/model/migel/product.rb:                             @limitation_text = LimitationText.new
src/model/migel/product.rb:                     def delete_limitation_text
src/model/migel/product.rb:                             if(lt = @limitation_text)
src/model/migel/product.rb:                                     @limitation_text = nil
src/model/migel/product.rb:        [ group, group.limitation_text, @subgroup, @subgroup.limitation_text,
src/model/migel/product.rb:          @product_text, self, @limitation_text ].compact.each { |item|
src/model/migel/subgroup.rb:                    attr_reader :products, :code, :limitation_text, :subgroup
src/model/migel/subgroup.rb:                            @limitation_text.odba_delete unless(@limitation_text.nil?)
src/model/migel/subgroup.rb:                    def create_limitation_text
src/model/migel/subgroup.rb:                            @limitation_text = LimitationText.new
src/model/migel/subgroup.rb:                    def delete_limitation_text
src/model/migel/subgroup.rb:                            if(lt = @limitation_text)
src/model/migel/subgroup.rb:                                    @limitation_text = nil
src/model/package.rb:           def limitation_text
src/model/package.rb:                   @sl_entry.limitation_text unless @sl_entry.nil?
src/model/registration.rb:              def limitation_text_count
src/model/registration.rb:                              inj + seq.limitation_text_count
src/model/sequence.rb:          def limitation_text
src/model/sequence.rb:                          if(txt = package.limitation_text)
src/model/sequence.rb:          def limitation_text_count
src/model/sequence.rb:                          package.limitation_text
src/model/slentry.rb:           attr_reader :limitation_text
src/model/slentry.rb:           def create_limitation_text
src/model/slentry.rb:                   @limitation_text = LimitationText.new
src/model/slentry.rb:           def delete_limitation_text
src/model/slentry.rb:                   @limitation_text = nil
src/plugin/analysis.rb:                         lim_ptr = pos.pointer + :limitation_text
src/plugin/analysis.rb:                 elsif(lim = pos.limitation_text)
src/plugin/bsv_xml.rb:            lim_ptr = @pointer + :limitation_text
src/plugin/bsv_xml.rb:                  :updated_sl_entries, :created_limitation_texts,
src/plugin/bsv_xml.rb:                  :deleted_limitation_texts, :updated_limitation_texts,
src/plugin/bsv_xml.rb:        @created_limitation_texts = 0
src/plugin/bsv_xml.rb:        @deleted_limitation_texts = 0
src/plugin/bsv_xml.rb:        @updated_limitation_texts = 0
src/plugin/bsv_xml.rb:              txt_ptr = sl_ptr + :limitation_text
src/plugin/bsv_xml.rb:                if sl_entry.limitation_text
src/plugin/bsv_xml.rb:                  @deleted_limitation_texts += 1
src/plugin/bsv_xml.rb:                if sl_entry.limitation_text
src/plugin/bsv_xml.rb:                  @updated_limitation_texts += 1
src/plugin/bsv_xml.rb:                  @created_limitation_texts += 1
src/plugin/csv_export.rb:        :limitation, :limitation_points, :limitation_text, :lppv,
src/plugin/csv_export.rb:        :limitation, :limitation_points, :limitation_text, :lppv,
src/plugin/migel.rb:                                            if((lt = prd.limitation_text) && lt.revision < @revision)
src/plugin/migel.rb:                                    if((lt = sbg.limitation_text) && lt.revision < @revision)
src/plugin/migel.rb:                            if((lt = grp.limitation_text) && lt.revision < @revision)
src/plugin/migel.rb:                            desc_ptr = pointer + [:limitation_text]
src/plugin/migel.rb:                            lim_ptr = pointer + [:limitation_text]
src/plugin/migel.rb:                            lim_ptr = pointer + [:limitation_text]
src/state/drugs/limitationtexts.rb:     DIRECT_EVENT = :limitation_texts
src/state/drugs/limitationtexts.rb:                     seq.limitation_text
src/state/drugs/limitationtexts.rb:             "sequence_limitation_text"
src/state/global.rb:                            :limitation_texts                       =>      State::Drugs::LimitationTexts,
src/state/global.rb:                                    :limitation_text ] =>   State::Drugs::LimitationText,
src/state/global.rb:                                    :limitation_text ] =>   State::Migel::LimitationText,
src/state/global.rb:                                    :limitation_text ] =>  State::Migel::LimitationText,
src/state/global.rb:                                    :limitation_text ] => State::Migel::LimitationText,
src/util/oddbapp.rb:    def count_limitation_texts
src/util/oddbapp.rb:                    inj + reg.limitation_text_count
src/util/oddbapp.rb:    def limitation_text_count
src/util/oddbapp.rb:            @limitation_text_count ||= count_limitation_texts()
src/util/oddbapp.rb:        @limitation_text_count = count_limitation_texts()
src/util/oddbapp.rb:#trace_local_var('@limitation_text')
src/util/oddbapp.rb:#p reg.packages[1].limitation_text.to_s
src/util/oddbapp.rb:  p pack.limitation_text.to_s
src/util/oddbapp.rb:  p pack.limitation_text.to_s
src/util/oddbapp.rb:#p result.atc_classes[0].packages[0].limitation_text
src/util/oddbapp.rb:  #p pack.limitation_text
src/util/oddbapp.rb:  p pack.limitation_text.to_s
src/util/oddbapp.rb:p $masa.packages[1].limitation_text.to_s
src/util/oddbapp.rb:                                    ptr = pos.pointer + :limitation_text
src/util/persistence.rb:        :sponsor, :substance, :user, :limitation_text, :minifi, 
src/util/validator.rb:                  :limitation_text,
src/util/validator.rb:                  :limitation_texts,
src/view/additional_information.rb:                     def limitation_text(model, session=@session)
src/view/additional_information.rb:                             if(sltxt = model.limitation_text)
src/view/additional_information.rb:                                     @lookandfeel.lookup(:limitation_text))
src/view/additional_information.rb:                             pos = components.index(:limitation_text)
src/view/analysis/position.rb:          [0,6]           =>      :limitation_text,
src/view/analysis/position.rb:  def limitation_text(model)
src/view/analysis/position.rb:          description(model.limitation_text, :limitation)
src/view/centeredsearchform.rb:                         @session.app.limitation_text_count.to_s << '&nbsp;'
src/view/dataformat.rb:#pp model.limitation_text
src/view/drugs/centeredsearchform.rb:                   :limitation_texts)
src/view/drugs/csv_result.rb:      'limitation_texts'         => 0,
src/view/drugs/csv_result.rb:        if sl.limitation_text
src/view/drugs/csv_result.rb:   def limitation_text(pack)
src/view/drugs/csv_result.rb:           if((sl = pack.sl_entry) && (txt = sl.limitation_text))
src/view/drugs/csv_result.rb:      @counts['limitation_texts'] += 1
src/view/drugs/limitationtext.rb:               [0,0]   =>      :limitation_text_title,
src/view/drugs/limitationtext.rb:       def limitation_text_title(model, session)
src/view/drugs/limitationtext.rb:                       @lookandfeel.lookup(:limitation_text_title, 
src/view/drugs/limitationtexts.rb:              [0,0]   => :limitation_text,
src/view/drugs/limitationtexts.rb:              :limitation_text        =>      :ltext,
src/view/drugs/limitationtexts.rb:              [0,0]   => :title_limitation_texts,
src/view/drugs/limitationtexts.rb:      def title_limitation_texts(model)
src/view/drugs/limitationtexts.rb:                      @lookandfeel.lookup(:title_limitation_texts, 
src/view/drugs/limitationtexts.rb:                              @session.limitation_text_count)
src/view/drugs/limitationtexts.rb:      SNAPBACK_EVENT = :limitation_texts
src/view/drugs/package.rb:      if(@model.limitation_text)
src/view/drugs/package.rb:        hash_insert_row(components, [0,10], :limitation_text)
src/view/drugs/package.rb:    if(idx = components.index(:limitation_text))
src/view/drugs/package.rb:      def limitation_text(model, session=@session)
src/view/drugs/package.rb:              if(lim = model.limitation_text)
src/view/drugs/resultlist.rb:           :limitation_text                =>      'list',
src/view/drugs/resultlist.rb:           :limitation_text                =>      'th',
src/view/drugs/resultlist.rb:           :limitation_text        =>      :ltext,
src/view/migel/group.rb:                [0,2] => :limitation_text,
src/view/migel/group.rb:        def limitation_text(model)
src/view/migel/group.rb:                description(model.limitation_text, :limitation_text)
src/view/migel/limitationtext.rb:               if(lim = product.subgroup.limitation_text)
src/view/migel/limitationtext.rb:               if(lim = product.group.limitation_text)
src/view/migel/limitationtext.rb:               [0,0]   =>      :limitation_text_title,
src/view/migel/limitationtext.rb:       def limitation_text_title(model, session)
src/view/migel/limitationtext.rb:                       @lookandfeel.lookup(:limitation_text_title, 
src/view/migel/product.rb:              [0,5] => :limitation_text,
src/view/migel/product.rb:      def limitation_text(model)
src/view/migel/product.rb:              description(model.limitation_text, :limitation_text)
src/view/migel/result.rb:    :limitation_text      => 'list',
src/view/migel/result.rb:               :limitation_text => :nbsp,
src/view/migel/result.rb:       def limitation_text(model)
src/view/migel/result.rb:               if(sltxt = model.limitation_text)
src/view/migel/result.rb:               values = [limitation_text(item), nil, item.migel_code, nil,
src/view/migel/subgroup.rb:             [0,3] => :limitation_text,
src/view/migel/subgroup.rb:     def limitation_text(model)
src/view/migel/subgroup.rb:             description(model.limitation_text, :limitation_text)

Read bsv_xml.rb

Source code

• src/plugin/bsv_xml.rb

Notes

• BsvXmlPlugin class (src/plugin/bsv_xml.rb) is called from Updater class (src/util/updater.rb)
• Updater#update_bsv method and Updater#reconsider_bsv method call the BsvXmlPlugin class.
• update method does just downloading the latest file, XMLPublications-latest.zip. The other process is delegated to _update method.
• The followings in _update method call the other 'update_XXXX' methods

          updater = $~[1].gsub(/[A-Z]/u) do |match| "_" << match.downcase end
          entry.get_input_stream do |io| send('update' << updater, io) end

• There are some 'update_XXXX' methods with io as an argument in BsvXmlPlugin class
  1. update_generics
  2. update_it_codes
  3. update_preparations
• There is 'update_chapter' but this belongs to BsvXmlPlugin::Listner class and this looks not having io object as an argument.
• tag_end methods look not called in oddb.org codes

masa@masa ~/ywesee/oddb.org $ grep -r tag_end src
src/plugin/bsv_xml.rb:      def tag_end name
src/plugin/bsv_xml.rb:      def tag_end name
src/plugin/bsv_xml.rb:      def tag_end name

• I guess REXML::Document.parse_stream calls that method, but I am not sure
• I have to check REXML Parser library
• Listner classes are probably called from REXML Parser library
• tag_start and tag_end methods are defined in every Listner class
• tag_start and tag_end methods look important
• I guess these method are probably called during the parsing process of xml file

Study REXML Parser library

Links

• REXML Home http://www.germane-software.com/software/rexml/
• Rdoc http://www.germane-software.com/software/rexml_doc/
• Rdoc http://pub.cozmixng.org/~kou/rexml-doc-ja/ (japanese)
• Sample http://www.nslabs.jp/ruby-rexml.rhtml (japanese)

Trace bsv_xml.rb

Notes

• There are 3 Listener classes defined in bsv_xml.rb
• Each class is corresponding to each xml file.
  1. PreparationsListener => Preparation.xml
  2. ItCodesListener => ItCodes.xml
  3. GenericsListener => Generics.xml
• In each listener class, tag_start and tag_end methods are defined, which are called from REXML parser library when reading a new tag.
• The tag_end method in PreparationListener class is the main part (and the longest method).
• The grabbing point of limitation description in PreparationListener is here: http://scm.ywesee.com/?p=oddb.org/.git;a=blob;f=src/plugin/bsv_xml.rb;h=220a65009827cbf2cc95067bc7e302d94ef6490d;hb=HEAD#l573

         if @in_limitation
           if @lim_data # we are within a Package
             chp = Text::Chapter.new
             update_chapter chp, @html
             @lim_data.store key, chp
           else
             @lim_texts.each_value do |text_data|
               chp = text_data[key] ||= Text::Chapter.new
               subheading = if @it_descriptions
                              [@itcode,
@it_descriptions[key]].compact.join(': ')
                            else
                              @name[key]
                            end
               update_chapter chp, @html, subheading
             end
           end

• According to this code, there look 2 kinds of Limitation Descriptions.
  1. Package (under Pack tag)
  2. the other (probably under Preparation tag)

Structure of xml file (regarding limitation)

There are 3 types of tag

1. Under Preparation tag
2. Under Pack tag
3. Under ItCode tag

1.

<Preparation>
 <Limitations>
  <Limitation>
  </Limitation>
 </Limitations>
</Preparation>

2.

<Preparation>
 <Packs>
  <Pack>
   <Limitations>
    <Limitation>
    </Limitation>
   </Limitations>
  </Pack>
 </Packs>
</Preparation>

3.

<Preparation>
 <ItCodess>
  <ItCode>
   <Limitations>
    <Limitation>
    </Limitation>
   </Limitations>
  </ItCode>
 </ItCodes>
</Preparation>

Experiment
src/plugin/bsv_xml.rb

        when 'SwissmedicNo5'
# masa
if @text.to_i == 56107 || @text.to_i == 55972
  print "SwissmedicNo5(end_tag): ", @text, "\n"
  @masa = true
else
  @masa = false
end

src/plugin/bsv_xml.rb

        when /^Description(..)$/u
          key = $~[1].downcase.to_sym
          if @in_limitation
            if @lim_data # we are within a Package
p 'yes @lim_data' if @masa
              chp = Text::Chapter.new
              update_chapter chp, @html
              @lim_data.store key, chp
            else
p 'no @lim_data' if @masa
              @lim_texts.each_value do |text_data|
                chp = text_data[key] ||= Text::Chapter.new
                subheading = if @it_descriptions
                               [@itcode, @it_descriptions[key]].compact.join(': ')
                             else
                               @name[key]
                             end
                update_chapter chp, @html, subheading
              end
            end
if @masa
  p @html
  print "@lim_data.length=", @lim_data.length, "\n"
  print "@lim_text.length=", @lim_text.length, "\n"
end

Result