view · edit · sidebar · attach · print · history

20110831-debug-bbmb_ch-update-search-migel

<< Masa.20110901-fulltext-search-migel | 2011 | Masa.20110830-debug-memory-leak-migel-overwriting-original-generic-flag-oddb_org >>

Debug missing 251 entry bbmb.ch
Debug miss-matching between file name and 010 entry
Debug 'FTP Fehler'
Updated migel search suspend

Goal/Estimate/Evaluation:

Debug bbmb.ch / 100% / 100%
Update migel search function / 90% / 80%

Milestones

~~bbmb.ch debug~~
migel search
1. create fulltext index table

Summary

http://masaywesee.wordpress.com/2011/08/31/debugged-missing-251-entry-in-xmlconv-and-updated-search-function-in-migel/

Commits

Updated for a new file format with customer ean code (globopharm.xmlconv.bbmb.ch)
http://scm.ywesee.com/?p=xmlconv/.git;a=commit;h=f93230d743458f111cdeb4a567b9642d38738897 (xmlconv)

ToDo

Refactor Group, Subgroup, Migelid because some code in the classes are totally same
Check Group and Subgroup de and fr view
Testcases oddb.org, ~~swissindex.rb~~, migel

Debug missing 251 entry bbmb.ch

Missing File Element 251 - bbmb.ch

Confirm

Run

masa@masa ~/ywesee/bbmb.ch $ bin/bbmbd
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ bin/xmlconvd config="etc/xmlconv.yml"

Access

http://masapharm.xmlconv.bbmb.ch/
Test ftp server

For example

http://masapharm.xmlconv.bbmb.ch/de/transaction/state_id/70215422427620/transaction_id/652

...
201:CU
202:7601001315747
201:BY
202:1075
201:DP
202:7601001315747
220:xxxx
221:xxxx
222:xxxx
223:xxxx
225:6593
231:xxxx
237:61
238:1
300:4
301:20110727
...

Note

In fact, there is not '251' entry

Experiment

convert_test_pharmacieplus_to_bdd_i2.rb

$: << "./lib"

require 'lib/conversion/propharma_bdd'
require 'lib/conversion/pharmacieplus_bdd'
require 'lib/conversion/bdd_csv'
require 'lib/conversion/bdd_i2'

# PharmaciePlusBdd (xml input file) -> BddI2 (formatted txt file)
reader = XmlConv::Conversion::PharmaciePlusBdd
input  = reader.parse(File.read('input.xml'))
model  = reader.convert(input)
writer = XmlConv::Conversion::BddI2
output = writer.convert(model)
print output.to_s

Run

masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ruby -I lib convert_test_pharmacieplus_bdd_i2.rb

Result

201:CU
202:7601001315747
201:BY
202:1075
201:DP
202:7601001315747
220:xxxx
221:xxxx
222:xxxx
223:xxxx
225:6593
231:secretariat@pharmacieplus.ch
237:61
238:1
300:4
301:20110831
...

Note

There is not '251' entry
Reader: PharmaciePlusBdd: 'lib/conversion/pharmacieplus_bdd'
Writer: BddI2: 'lib/conversion/bdd_i2'

Note

251 entry is described in lib/xmlconv/i2/order.rb

masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ grep -r 251 lib/
lib/xmlconv/i2/order.rb:          output << sprintf("250:ADE\n251:%s\n", @ade_id)

But this library is not required in lib/conversion/bdd_i2.rb

masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ head lib/conversion/bdd_i2.rb 
require 'xmlconv/i2/document'
require 'xmlconv/i2/address'
require 'xmlconv/i2/position'

Experiment

lib/conversion/bdd_i2.rb

        def _doc_add_delivery(doc, delivery, sender_id='YWESEE')
          order = I2::Order.new
          order.sender_id = sender_id
          # customer_id is in reality the delivery_id assigned by the
          # customer - the slight confusion is due to automatic naming
          #transaction_id = delivery.customer_id
          #order.ade_id = order.delivery_id = transaction_id
          #doc.header.transaction_id = sprintf(transaction_id.to_s.rjust(8, '0'))
          transaction_id = delivery.customer_id
          order.ade_id = order.delivery_id = transaction_id
          doc.header.transaction_id = sprintf(transaction_id.to_s.rjust(8, '0'))

Run

masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ruby convert_test_pharmacieplus_to_bdd_i2.rb

Result

 001:7601001000681
 002:ORDERX
 003:220
 010:KD7601001315747_00005286.dat
 100:YWESEEPP
 101:5286
 201:CU
 202:7601001315747
 201:BY
 202:1075
 201:DP
 202:7601001315747
 220:xxxx
 221:xxxx
 222:xxxx
 223:xxxx
 225:6593
 231:xxxx
 237:61 
 238:1
 250:ADE
 251:5286
 300:4
 301:20110831

Note

In order to show '251' entry, @ade_id data is necessary, which is set in lib/conversion/bdd_i2.rb
It was commented out in the last update. see: http://dev.ywesee.com/wiki.php/Masa/20110727-testcases-bbmbCh-change-output-filename-xmlconvBbmbCh#FilenameXmlconv
I cannot remember why this was commented out

Check it on ftp server

Run

masa@masa ~/ywesee/bbmb.ch $ bin/bbmbd
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ bin/xmlconvd config="etc/xmlconv.yml"

Setup

copy input.xml to polling directory, "/home/masa/ywesee/globopharm.xmlconv.bbmb.ch/xml_orders"
wait one minute
check ftp server and xmlconv server (http://masapharm.xmlconv.bbmb.ch/)

masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ls xml_orders/
input.xml
...
(one minute later)
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ls xml_orders/
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $

Result

'251' entry is shown on xmlconv server but xmlconv server says 'FTP Fehler'

Experiment

lib/xmlconv/util/application.rb

      def _execute(transaction)
p "getin _execute"
        transaction.transaction_id = next_transaction_id
p "B"
        transaction.execute
p "C"
        #transaction.postprocess
        ret = transaction.postprocess
p "D"
ret
      rescue Exception => error

Create '/IN/Temp' directory on the FTP server

Result

"getin poll_sources"
"getin load_sources"
CONFIG.polling_file = "/home/masa/ywesee/globopharm.xmlconv.bbmb.ch/etc/polling.yml"
source.class = XmlConv::Util::PollingMission
transaction.class = XmlConv::Util::Transaction
"getin _execute"
"B"
"C"
...

Result

The result files are shown on the FTP server

IMPORTANT

The postprocess needs '/IN/Temp' directory on the FTP server
This setting is written in etc/polling.yml file

tmp_destination: "/IN/Temp"

But actually, 'D' is not shown, which means 'transaction.postprocess' is not working
Also, 'FTP Fehler' is shown on the xmlconv server (http://masapharm.xmlconv.bbmb.ch/de)

Debug miss-matching between file name and 010 entry

Fwd: PharmaciePlus File

z.B.

masa@masa ~/work $ grep -r KD7601001005631_00005286 KD7601001005631_00005286012.dat 
010:KD7601001005631_00005286.dat

Note

In fact, a file name have 3 digits more are added to the 010 entry

suspend

Debug 'FTP Fehler'

Problem

FTP transaction succeeds but the status on xmlconv server becomes 'Fehler'

Experiment

(Run/Stop) bin/bbmbd
Run bin/xmlconvd

Result

The status becomes 'ftp_ok' if bin/bbmbd is running at the same time
But the status becomes 'Fehler' if bin/bbmbd is NOT running at the same time

Note

If the 3 lines are commented out above, the transaction_id (suffix of file name) becomes 'date+time+index' as I updated on 27th Jul.
If they are commented in again, the transaction_id becomes the old format

Experiment

lib/conversion/bdd_i2.rb

        def _doc_add_delivery(doc, delivery, sender_id='YWESEE')
          order = I2::Order.new
          order.sender_id = sender_id
          # customer_id is in reality the delivery_id assigned by the
          # customer - the slight confusion is due to automatic naming
          transaction_id = delivery.customer_id
          order.ade_id = order.delivery_id = transaction_id
          #doc.header.transaction_id = sprintf(transaction_id.to_s.rjust(8, '0'))
          order.add_date(I2::Date.from_date(::Date.today, :order, :order))

Result

file name becomes with date+time+index, and 251 entry is also shown

Note

ade_id is necessary to show 251 entry, but doc.header.transaction_id is not necessary to make a suffix of file name like date+time+index

Commit

Updated migel search

Problem

At the moment, the search function searchs only one of Group, Subgroup or Migelid
It should search all of them at them same time and show all the results

Note

src/util/oddbapp.rb#search_migel_products should be updated

    def search_migel_products(query, lang)
      migel_code = if query =~ /(\d){9}/
                     query.split(/(\d\d)/).select{|x| !x.empty?}.join('.')
                   elsif query =~ /(\d\d\.){4}\d/
                     query
                   end
      if migel_code
         MIGEL_SERVER.migelid.search_by_migel_code(migel_code)
      else
        search_method = 'search_by_name_' + lang.downcase.to_s
        MIGEL_SERVER.search_migelid_by_name(query, lang.downcase.to_s)
      end
    end

lib/migel/util/server.rb#search_migel_by_name

      def search_migelid_by_name(query, lang)
        search_method = 'search_by_name_' + lang
        result = []

        if groups = Migel::Model::Group.send(search_method, query) and !groups.empty?
          groups.each do |group|
            result.concat group.subgroups.collect{|sg| sg.migelids}.flatten
          end
        end
        if subgroups = Migel::Model::Subgroup.send(search_method, query) and !subgroups.empty?
          result.concat subgroups.collect{|sg| sg.migelids}.flatten
        end
        result.concat Migel::Model::Migelid.send(search_method, query)
        ODBA::DRbWrapper.new(result.uniq)
      end

Run

bin/oddbd
bin/currencyd
bin/migeld

Access (search 'material')

http://oddb.masa.org/de/gcc/search/zone/migel/search_query/material#best_result

Result

Online

Local

Note

The reason of the difference above is as follows
1. Online server searches the query not only names of Group, Subgroup, Migelid but also @description data by using 'migel_fulltext_index_de(fr)' table
2. Local server searches only the names of Group, Subgoup, and Migelid.

Create a fulltext table

Question

How to create fulltext table as well as oddb.org

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110831-debug-bbmb_ch-update-search-migel

Debug missing 251 entry bbmb.ch

Confirm

Check it on ftp server

Debug miss-matching between file name and 010 entry

Debug 'FTP Fehler'

Updated migel search

ywesee Developer-Wiki Dieses Wiki richtet sich an alle ywesee-Entwickler

About

EBPS

Bbmb

ODBA

Oddb

Rpdf2txt

YDPM

YDIM

XmlConv

20110831-debug-bbmb_ch-update-search-migel

Debug missing 251 entry bbmb.ch

Confirm

Check it on ftp server

Debug miss-matching between file name and 010 entry

Debug 'FTP Fehler'

Updated migel search

ywesee Developer-Wiki
Dieses Wiki richtet sich an alle ywesee-Entwickler