view · edit · sidebar · attach · print · history

20110107-debug-bag-list-ticket250

<< | Index | >>

Check BAG list
Check 'SB' updating process one by one

Goal

Debug BAG list / 70%

Milestones

~~Check BAG list~~ 8:30
First, focus only on 7680545770922
- ~~Diff the downloaded xml file and online xml file~~ 8:30
- ~~Check it locally too~~ 8:30
- Check the process to update the flag 'SB' from Preparations.xml
- Understand the process (difference between '20%' and 'k.A')
Check the other products
(Make a script to check all the cache data again)
(Make a test-case)

Summary

http://masay.ch/2011/01/07/debug-bag-list/

Commits
ToDo Tomorrow
Keep in Mind

Check BAG list

Reference

Ticket #250

Check the online latest xml file data and online ch.oddb.org data

For example

'SB' of 54577092 should be '20%'
But it is not on ch.oddb.org

ToDo

First, focus only on 7680545770922
Diff the downloaded xml file and online xml file
Check it locally too
Check the process to update the flag 'SB' from Preparations.xml
Understand the process (difference between '20%' and 'k.A')
Check the other products
(Make a script to check all the cache data again)

Diff the downloaded xml on production server and the online xml file

I confirmed they are totally same

Note

Namely, the updating process has a problem

Check the data locally

The 'SB' flag of the product 7680545770922 'Temodal' is not '20%'

Check the process to update the flag 'SB' from Preparations.xml

My hypothesis

The updating is not the full update but the diff update
In other words, if there is no difference between the current data and the new data, nothing is done

Consideration

There should be the comparison process somewhere, and I should check carefully the comparison process
There is a possibility that the cache data is the same with the xml file data but the shown data is different
I should check what kind of the cache data is, if the shown data and original data are stored separately or not.

Follow the updating process

Updater#run
Updater#update_bsv
BsvXmlPlugin#update
BsvXMLPlugin#_update
BsvXmlPlugin#update_preparations
PreparationsListener, REXML#parse_stream

Note

PreparationsListener class and REXML class are correlated
It is difficult to trace the flow only by reading the code of oddb.org after this point
I guess REXML.parse_stream method calls methods of Listener class which is given as an argument to the parse_stream method
- (probably these libraries use 'Template method' design pattern)
I should look at data step by step after this
The important part may be 'tag_end' method in the Listener class

Preparation of experiment data

Delete the other data than 7680545770922 from Preparations.xml

Attach:Preparations.20110107.xml.txt

Zip Preparations.xml to XMLPublications.zip

masa@masa ~/work $ zip XMLPublications.zip Preparations.xml
  adding: Preparations.xml (deflated 77%)

Attach:XMLPublications.20110107.zip

Modify a code a little bit for experiment

src/plugin/bsv_xml.rb#update

      #path = download_file(target_url, save_dir, file_name)
      path = "/home/masa/work/XMLPublications.zip"

Run

bin/oddbd
ext/meddata/bin/meddatad
currencyd
(yusd)

Run updater (update_bsv)

From bin/admin

masa@masa ~/ywesee/oddb.org $ bin/admin 
ch.oddb> Updater.new(self).update_bsv

Result

Email ch.ODDB.org Report - SL-Update (XML) - 01/2011

Created SL-Entries                                            0
Updated SL-Entries                                            0
Deleted SL-Entries                                         8291
Created Limitation-Texts                                      0
Updated Limitation-Texts                                      0
Deleted Limitation-Texts                                      0

Duplicate Registrations in SL 07.01.2011                      0
Package-Data was completed from SL                            0
SMeX/SL-Differences (Registrations) 07.01.2011                0
SMeX/SL-Differences (Packages) 07.01.2011                     1
Critical Pharmacodes BAG-XML 07.01.2011                       0
Missing Swissmedic-Codes in SL 07.01.2011                     0
Missing Pharmacodes in SL 07.01.2011                          0
Missing Swissmedic-Codes in SL (out of trade) 07.01.2011      0
Unknown Packages in SL 07.01.2011                             0
Unknown Registrations in SL 07.01.2011                        0
Unknown Packages in SL (out of trade) 07.01.2011              0
Packungen in der ODDB Total: 23259
Packungen ohne Pharmacode: 8625
- ausser Handel: 7790
- inaktive Registration: 671
- noch nicht auf MedWin: 164

Check on browser

before

after

Check 'SB' updating process one by one

Experiment

src/plugin/bsv_xml.rb#end_tag

     def tag_end name
...
        when 'FlagSB20'
p "getin FlagSB20"
print "@text=", @text, "\n"
          @pac_data.store :deductible, @text == 'Y' ? :deductible_o : :deductible_g

Restart bin/oddbd

Run update_bsv

masa@masa ~/ywesee/oddb.org $ bin/admin 
ch.oddb> Updater.new(self).update_bsv

Result

"getin FlagSB20"
@text=Y

Note

The data is correctly read as expected
Next, trace further

But

I cannot find the place where the @pac_data is used after the code above

Experiment

src/plugin/bsv_xml.rb#end_tag

      def tag_end name
        case name
        when 'Pack'
p "getin tag_end#Pack"
print "@data="
p @data

Restart bin/oddbd

Run update_bsv

Result

 @data={:price_exfactory=>#<ODDB::Util::Money:0x7faa6be97650 @credits=3923, @mutation_code="SLAUFNAHME", @amount=39.23, @origin="http://bag.e-mediat.net/SL2007.Web.External
 /File.axd?file=XMLPublications.zip (07.01.2011)", @valid_from=Thu Jul 01 00:00:00 +0200 2010, @type="exfactory", @country="CH", @authority=:sl>, :ikscat=>"A", :pharmacode=>"4621130", 
 :narcotic=>false, :sl_generic_type=>:original, :price_public=>#<ODDB::Util::Money:0x7faa6be82c78 @credits=6140, @mutation_code="SLAUFNAHME", @amount=61.4, @origin="http://bag.e-mediat.net
 /SL2007.Web.External/File.axd?file=XMLPublications.zip (07.01.2011)", @valid_from=Thu Jul 01 00:00:00 +0200 2010, @type="public", @country="CH", @authority=:sl>, 
 :deductible=>:deductible_o}

Note

':deductible' is certainly :deductible_o (which means FlagSB20 is 'Y')

Hypothesis

:deductible value is not updated

Experiment

src/plugin/bsv_xml.rb#tag_end

      def tag_end name
        case name
        when 'Pack'

p "getin tag_end#Pack"
print "1. @data="
pp @data
          if @pack.nil? && @completed_registrations[@iksnr] && !@out_of_trade
            @deferred_packages.push({
              :ikscd    => @ikscd,
              :sequence => @seq_data,
              :package  => @data,
              :sl_entry => @sl_data,
              :lim_text => @lim_data,
              :size     => @size,
            })
          elsif @pack && !@conflict && !@duplicate_iksnr
            @report.store :pharmacode_oddb, @pack.pharmacode
            if seq = @pack.sequence
              @app.update seq.pointer, @seq_data, :bag
            end
            pold = @pack.price_public
            pnew = @data[:price_public]
            unless pold && pnew
              pold = @pack.price_exfactory
              pnew = @data[:price_exfactory]
            end
            if pold && pnew
              if pold > pnew
                flag_change @pack.pointer, :price_cut
              elsif pold < pnew
                flag_change @pack.pointer, :price_rise
              end
            end

print "@pack="
pp @pack
print "2. @data="
pp @data

            ## don't take the Swissmedic-Category unless it's missing in the DB
            @data.delete :ikscat if @pack.ikscat
            @app.update @pack.pointer, @data, :bag
            @sl_entries.store @pack.pointer, @sl_data
            @lim_texts.store @pack.pointer, @lim_data
          end

Restart bin/oddbd

Run update_bsv

Result

 1. @data={:price_exfactory=>
  #<ODDB::Util::Money:0x7f2ed8186290
   @amount=39.23,
   @authority=:sl,
   @country="CH",
   @credits=3923,
   @mutation_code="SLAUFNAHME",
   @origin=
    "http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip (07.01.2011)",
   @type="exfactory",
   @valid_from=Thu Jul 01 00:00:00 +0200 2010>,
 :ikscat=>"A",
 :pharmacode=>"4621130",
 :narcotic=>false,
 :sl_generic_type=>:original,
 :price_public=>
  #<ODDB::Util::Money:0x7f2ed8174040
   @amount=61.4,
   @authority=:sl,
   @country="CH",
   @credits=6140,
   @mutation_code="SLAUFNAHME",
   @origin=
    "http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip (07.01.2011)",
   @type="public",
   @valid_from=Thu Jul 01 00:00:00 +0200 2010>,
 :deductible=>:deductible_o}

Note

Namely, the :deductible is certainly :deductible_o but this data is not used for the updating of package

Memo

<FlagSB20> tag is under <Preparation> layer
So, the updating process should be in 'Preparation' of 'tag_end' method, not 'Pack'
That is why the updating process does not run if there is not a difference regarding 'Pack'

What to do next?

Experiment

src/plugin/bsv_xml.rb#tag_end

            ## don't take the Swissmedic-Category unless it's missing in the DB
            @data.delete :ikscat if @pack.ikscat
            @app.update @pack.pointer, @data, :bag
            @sl_entries.store @pack.pointer, @sl_data
            @lim_texts.store @pack.pointer, @lim_data
          end
@app.update @pack.pointer, @data, :bag

          @pcode, @pack, @sl_data, @lim_data, @out_of_trade, @ikscd, @data,
            @size = nil
        when 'Preparation'

Note

This means the updating the package data forcibly

Restart bin/oddbd

Run update_bsv

Result

Note

Good

Comparison the :deductible value both in cache and @pac_data

Experiment (check the cache data)

src/plugin/bsv_xml.rb#tag_end

            ## don't take the Swissmedic-Category unless it's missing in the DB
            @data.delete :ikscat if @pack.ikscat
            @app.update @pack.pointer, @data, :bag
            @sl_entries.store @pack.pointer, @sl_data
            @lim_texts.store @pack.pointer, @lim_data
          end

print "@data[:deductible]="
pp @data[:deductible]
print "@pack.deductible="
pp @pack.deductible
pack_ptr = @pack.pointer
pack = pack_ptr.resolve(@app)
print "pack.deductible="
pp pack.deductible

Restart bin/oddbd

Run update_bsv

Result

 @data[:deductible]=:deductible_o   #read data from xml file
 @pack.deductible=nil               #cache data
 pack.deductible=nil                #cache data

Note

I can use this data for the comparison

Experiment

src/plugin/bsv_xml.rb#tag_end


            ## don't take the Swissmedic-Category unless it's missing in the DB
            @data.delete :ikscat if @pack.ikscat
            @app.update @pack.pointer, @data, :bag
            @sl_entries.store @pack.pointer, @sl_data
            @lim_texts.store @pack.pointer, @lim_data

          elsif @pack

            if @pack.deductible != @pac_data[:deductible]
              @app.update @pack.pointer, @data, :bag
            end

          end

Restart bin/oddbd

Run update_bsv

Result

Note

Looks good

Final check locally

Stop all daemons
Restore the current cache data

~/ywesee/oddb.org $ sudo -u postgres dropdb oddb.org
~/ywesee/oddb.org $ sudo -u postgres createdb -E UTF8 -T template0 oddb.org
~/ywesee/oddb.org $ zcat data/sql/oddb.org.20110106.sql.gz |psql -U postgres oddb.org

Restart
- bin/oddbd
- ext/meddata/bin/meddatad
- currencyd
Run update_bsv with the latest XMLPublications.zip (include all the data)
- http://bag.e-mediat.net/SL2007.Web.External/File.axd?file=XMLPublications.zip

masa@masa ~/ywesee/oddb.org $ bin/admin
ch.oddb> Updater.new(self).update_bsv

Changed part src/plugin/bsv_xml.rb#tag_end

            ## don't take the Swissmedic-Category unless it's missing in the DB
            @data.delete :ikscat if @pack.ikscat
            @app.update @pack.pointer, @data, :bag
            @sl_entries.store @pack.pointer, @sl_data
            @lim_texts.store @pack.pointer, @lim_data

            elsif @pack
            if @pack.deductible != @pac_data[:deductible]
              @app.update @pack.pointer, @data, :bag
            end

          end

This is only for this test src/plugin/bsv_xml.rb#update

      #path = download_file(target_url, save_dir, file_name)
      path = "/home/masa/work/XMLPublications.zip"

Result

Note

Looks good

Test deleting a package from admin tool

suspend

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110107-debug-bag-list-ticket250

Check BAG list

Diff the downloaded xml on production server and the online xml file

Check the data locally

Check the process to update the flag 'SB' from Preparations.xml

Preparation of experiment data

Check 'SB' updating process one by one

Final check locally

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110107-debug-bag-list-ticket250

Check BAG list

Diff the downloaded xml on production server and the online xml file

Check the data locally

Check the process to update the flag 'SB' from Preparations.xml

Preparation of experiment data

Check 'SB' updating process one by one

Final check locally

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler