view · edit · sidebar · attach · print · history

20160819-debug-product-confusion-sandoz-xmlconv-bbmb-ch

<< 20160822-debug-product-confusion-sandoz-xmlconv-bbmb-ch-migrate-davaz-com | Index | 20160818-migrate-davaz-com >>

Summary

Debug confused product issue on sandoz.bbmb.ch

Commits / Patches / Pull Requests, Scripts

Add to_product method to ProductInfo [commit, bbmb]
Add `to_product` utility method [pull request, bbmb]
- Create script to fix data corruption
- Bump version 2.1.0
Create script to fix data corruption [pull request, sandoz.bbmb.ch]
- Add fix product script
Add article_number index as fix target [pull request, sandoz.bbmb.ch]
- Add article_number index as fix target

Index

Product confusion issue
- Debug product importer
- Debug order injection
Repair BBMB Product
- How to fix corrupted data
Fixed-result

Product confusion issue

A product ordered from xmlconv, as confused as other product.

sandoz.xmlconv.bbmb.ch:
/de/transaction/state_id/70154154923740/transaction_id/29340

bbmb.ch:
/de/order/order_id/4100602981-330

# sandoz.xmlconv.bbmb.ch
    <productOrderLine orderQuantity="3"
         roundUpForCondition="true" backLogDesired="true">
      <pharmaCode id="2508406" />
    </productOrderLine>

# sandoz.bbmb.ch
3	Citalopram Sandoz 40 mg Film Tbl 98

In Pcode (Pharmacode).

2508406 => 5423389

At Artikel.txt

# ordered product
44060691  Cip eco 500 mg Film Tbl 20  Cip eco 500 mg cpr pell 20  7680555250186 2508406 24.62

# wrong product
44045701  Citalopram Sandoz 40 mg Film Tbl 98 Citalopram Sandoz 40 mg cpr pell 98 7680559350127 5423389 114.2

Debug product importer

ProductImporter parse data correctly.

[1] 2.3.1-p112(#<BBMB::Util::ProductImporter>)> record
=> ["44060692", "Cip eco 750 mg Film Tbl 20", "Cip eco 750 mg cpr pell 20", "7680555250209", "2508412", "36.93\r\n"]
[2] 2.3.1-p112(#<BBMB::Util::ProductImporter>)> object = import_record(record)
=> #<BBMB::Model::Product:0x0055e6af45baf0
 @article_number="44060692",
 @backorder=false,
 @description=
  #<BBMB::Util::Multilingual:0x0055e6af45b820
   @canonical={:de=>"Cip eco 750 mg Film Tbl 20", :fr=>"Cip eco 750 mg cpr pell 20"},
   @synonyms=[]>,
 @ean13="7680555250209",
 @odba_id=489364,
 @odba_observers=[],
 @odba_persistent=true,
 @pcode="2508412",
 @price=#<BBMB::Util::Money:0x0055e6af45a2b8 @credits=3693>>

Debug order injection

Xmlconv parse and throw order to bbmb. correctly.

# In xmlconvd, before inject_order
[3] 2.3.1-p112(XmlConv::PostProcess::Bbmb2)> order
=> [{:pcode=>"3458461", :quantity=>1},
 {:pcode=>"5195161", :quantity=>1},
 {:pcode=>"6211896", :quantity=>1},
 {:pcode=>"4470932", :quantity=>2},
 {:pcode=>"2508398", :quantity=>3},
 {:pcode=>"2508406", :quantity=>3},
 {:pcode=>"4237992", :quantity=>2}]

BBMB server receives order, correctly.

[1] 2.3.1-p112(#<BBMB::Util::Server>)> products
=> [{:pcode=>"3458461", :quantity=>1},
 {:pcode=>"5195161", :quantity=>1},
 {:pcode=>"6211896", :quantity=>1},
 {:pcode=>"4470932", :quantity=>2},
 {:pcode=>"2508398", :quantity=>3},
 {:pcode=>"2508406", :quantity=>3},
 {:pcode=>"4237992", :quantity=>2}]

But it looks like a little bit strange... This is not actual Product. (only this, Position)
What is this?

[1] 2.3.1-p112(#<BBMB::Util::Server>)> info
=> {:pcode=>"2508406", :quantity=>3}
[2] 2.3.1-p112(#<BBMB::Util::Server>)> product
=> #<BBMB::Model::Order::Position:0x007f5095f552c0
 @odba_id=506359,
 @odba_observers=[],
 @odba_persistent=true,
 @price_effective=#<BBMB::Util::Money:0x007f5095f54938 @credits=11423>,
 @product=
  #<BBMB::Model::ProductInfo:0x007f5095f551f8
   @article_number="44045701",
   @backorder=false,
   @backorder_date=nil,
   @catalogue1=#<BBMB::Util::Multilingual:0x007f5095f54c08 @canonical={}, @synonyms=[]>,
   @catalogue2=#<BBMB::Util::Multilingual:0x007f5095f54d98 @canonical={}, @synonyms=[]>,
   @catalogue3=#<BBMB::Util::Multilingual:0x007f5095f54fa0 @canonical={}, @synonyms=[]>,
   @description=
    #<BBMB::Util::Multilingual:0x007f5095f54b40
     @canonical={:de=>"Cip eco 500 mg Film Tbl 20", :fr=>"Cip eco 500 mg cpr pell 20"},
     @synonyms=[]>,
   @ean13="7680555250186",
   @expiry_date=nil,
   @l1_price=nil,
   @l1_qty=nil,
   @l2_price=nil,
   @l2_qty=nil,
   @l3_price=nil,
   @l3_qty=nil,
   @l4_price=nil,
   @l4_qty=nil,
   @l5_price=nil,
   @l5_qty=nil,
   @l6_price=nil,
   @l6_qty=nil,
   @partner_index=nil,
   @pcode="2508406",
   @price=#<BBMB::Util::Money:0x007f5095f55130 @credits=2462>,
   @status=nil,
   @vat=nil>,
 @quantity=2>

Normal other product is like this:

[3] 2.3.1-p112(#<BBMB::Util::Server>)> info
=> {:pcode=>"4237992", :quantity=>2}
[4] 2.3.1-p112(#<BBMB::Util::Server>)> product
=> #<BBMB::Model::Product:0x007f5096888900
 @article_number="44048860",** [[#how-to-fix-corrupted-data|How to fix corrupted data]]
 @backorder=false,
 @description=
  #<BBMB::Util::Multilingual:0x007f50968886a8
   @canonical={:de=>"Perindopril Sandoz 4mg Tbl 30", :fr=>"Perindopril Sandoz 4 mg cpr 30"},
   @synonyms=[]>,
 @ean13="7680583330041",
 @odba_id=400705,
 @odba_observers=[],
 @odba_persistent=true,
 @pcode="4237992",
 @price=#<BBMB::Util::Money:0x007f5096888888 @credits=1090>>

This might be strange...

[6] 2.3.1-p112(#<BBMB::Util::Server>)> order.positions.map {|ps| ps.product.class }
=> [BBMB::Model::Product,
 BBMB::Model::Product,
 BBMB::Model::Product,
 BBMB::Model::Product,
 BBMB::Model::Product,
 BBMB::Model::Order::Position,  #=> this is the cosfused product.
 BBMB::Model::Product]

When I post same order request via sandoz.xmlconv.bbmb.ch, In my local, order is fine.

I think this issue caused by data corruption of ODBA using Ruby's object id.

Repair BBMB Product

According my unterstand, the updating by updater script and the saving of order from customer are processed at same time.
This issue is caused, because ODBA is not thread safe for it's id using object_id.
If ruby uses same object_id in both process, object might be confused.

Then, at first, I've decide to fix this data corruption above.
This is not fundamental solution, but I think this is related in this product data confusion (or this might be also affected result).
But, how to load all products to check themself like oddb.org? (in bin/admin)

oddb.org loads registrations from cached index at boot sequence.
Then I've tried to fetch all products from index in bbmb's bin/admin.

In oddb.org, load all objects via this registration, which is loaded from cache.

# bin/admin
ch.bbmb.sandoz> p ODBA.cache.indices.keys.to_a
-> Array

# output
[
"bbmb_model_product_ean13", "bbmb_model_product_pcode", "bbmb_model_customer_ean13",
"bbmb_model_customer_email", "bbmb_model_product_catalogue1", "bbmb_model_product_catalogue2",
"bbmb_model_product_description", "bbmb_model_customer_customer_id", "bbmb_model_product_article_number",
"ydim_debitor_name", "ydim_debitor_email", "ydim_invoice_status",
"ydim_debitor_unique_id", "ydim_invoice_unique_id", "ydim_autoinvoice_unique_id"
]

Links

I've found a way to fetch all products from index! (like oddb's @registrations)

ch.bbmb.sandoz> ODBA.cache.retrieve_from_index('bbmb_model_product_pcode', '').length
-> 1001

And I've found 8 strange products in database.

ch.bbmb.sandoz> p ODBA.cache.retrieve_from_index('bbmb_model_product_pcode', '').select { \
|t| !t.is_a?(BBMB::Model::Product) }.length
-> 8

ch.bbmb.sandoz> p ODBA.cache.retrieve_from_index('bbmb_model_product_pcode', '').select { \
|t| !t.is_a?(BBMB::Model::Product) }.map {|t| t.class }
-> [
BBMB::Model::Order::Position,
BBMB::Model::Order::Position,
BBMB::Model::Order::Position,
Array,
Array,
BBMB::Model::Order::Position,
BBMB::Model::Order::Position,
BBMB::Model::Order::Position
]

Product model has to_info, but ProductInfo model has not to_product.
Then I've adde this util method to fix wrong model objects.

Learn about ODBA' data structure.

index (z.B. bbmb_model_product_pcode) is defined as normal table.
origin is previouse object' id (Maybe, at initialization, same as target_id)
Current entry' id is described as target

bbmb_sandoz=> SELECT * from bbmb_model_product_pcode WHERE search_term = '6479647';
 origin_id | search_term | target_id 
-----------+-------------+-----------
    509907 | 6479647     |    509907

Therefore, solution is:

ch.bbmb.sandoz> ti = ODBA.cache.retrieve_from_index('bbmb_model_product_pcode', ''); \
w = ti.select {|t| !t.is_a?(BBMB::Model::Product) }.first; \
t = w.product.to_product; \
t.odba_store; \
td = ODBA.cache.indices.fetch('bbmb_model_product_pcode'); \
td.do_update_index(t.odba_id, '6479647', t.odba_id); \
-> 1

# right result
ch.bbmb.sandoz> p BBMB::Model::Product.find_by_pcode('6479647').class
-> BBMB::Model::Product

# it looks like LIKE query (LIKE % '')
ti = ODBA.cache.retrieve_from_index('bbmb_model_product_pcode', '')

Array...

Some corrupted data are Array objects, and these array has multiple products.

ch.bbmb.sandoz> ti = ODBA.cache.retrieve_from_index('bbmb_model_product_pcode', ''); \
w = ti.select {|t| !t.is_a?(BBMB::Model::Product) && t.is_a?(Array) }.first; \
p w.odba_id
-> 506573

Therefore, we have to fix index to point right single product.

How to fix corrupted data

1. Apply this change to bbmb

Add to_product method to ProductInfo

2. Run, fix_product script at sandoz.bbmb.ch

Fix fix string encoding script

% bundle exec ruby fix_product config=/var/www/sandoz.bbmb.ch/etc/config.yml

3. Run updater job with product data (csv)

4. Restart bbmbd.

Fixed result

This is result log on production

script: fix_product

fixing index: bbmb_model_product_pcode
EE......
fixing index: bbmb_model_product_ean13
...EE...

NOTE: if "E" appeared in output, then you should also run updater script

The following product is missing. (this must be imported by updater or something)

2508398 (pharmacode)
6479653 (pharmacode)
7680555250162 (ean13)
7680656510028 (ean13)

This meands these 2 products:

44060690  Cip eco 500 mg Film Tbl 10  Cip eco 500 mg cpr pell 10  7680555250162 2508398 12.44
44065710  Fulvestrant Sandoz 2 Fertigspritzen 250m  Fulvestrant Sandoz 2 seringue pr?remplie  7680656510028 6479653 703.76

I've found one product of these products

ch.bbmb.sandoz> p BBMB::Model::Customer.find_by_customer_id('4100602981').orders.find {|o| \
o.order_id =~ /330$/ }.positions.map {|s| s.product.to_product.pcode }
-> ["6211896", "5423389", "3458461", "4237992", "5195161", "4470932", "2508398"]

# How to fix this, manually.
ch.bbmb.sandoz> p BBMB::Model::Customer.find_by_customer_id('4100602981').orders.find {|o| \
o.order_id =~ /330$/ }.positions.find {|s| s.product.to_product.pcode == "2508398" }.product.to_product.odba_store
->
ch.bbmb.sandoz> p BBMB::Model::Product.find_by_pcode('2508398').pcode
-> 2508398

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20160819-debug-product-confusion-sandoz-xmlconv-bbmb-ch

Summary

Commits / Patches / Pull Requests, Scripts

Index

Product confusion issue

Debug product importer

Debug order injection

Repair BBMB Product

Links

Learn about ODBA' data structure.

Array...

How to fix corrupted data

Fixed result

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20160819-debug-product-confusion-sandoz-xmlconv-bbmb-ch

Summary

Commits / Patches / Pull Requests, Scripts

Index

Product confusion issue

Debug product importer

Debug order injection

Repair BBMB Product

Links

Learn about ODBA' data structure.

Array...

How to fix corrupted data

Fixed result

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler