view · edit · sidebar · attach · print · history

20110831-debug-bbmb_ch-update-search-migel

<< Masa.20110901-fulltext-search-migel | 2011 | Masa.20110830-debug-memory-leak-migel-overwriting-original-generic-flag-oddb_org >>


  1. Debug missing 251 entry bbmb.ch
  2. Debug miss-matching between file name and 010 entry
  3. Debug 'FTP Fehler'
  4. Updated migel search suspend

Goal/Estimate/Evaluation:

  • Debug bbmb.ch / 100% / 100%
  • Update migel search function / 90% / 80%
Milestones
  • bbmb.ch debug
  • migel search
    1. create fulltext index table
Summary
Commits
ToDo
  • Refactor Group, Subgroup, Migelid because some code in the classes are totally same
  • Check Group and Subgroup de and fr view
  • Testcases oddb.org, swissindex.rb, migel

Debug missing 251 entry bbmb.ch

Email

Confirm

Run

  • masa@masa ~/ywesee/bbmb.ch $ bin/bbmbd
  • masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ bin/xmlconvd config="etc/xmlconv.yml"

Access

For example

...
201:CU
202:7601001315747
201:BY
202:1075
201:DP
202:7601001315747
220:xxxx
221:xxxx
222:xxxx
223:xxxx
225:6593
231:xxxx
237:61
238:1
300:4
301:20110727
...

Note

  • In fact, there is not '251' entry

Experiment

  • convert_test_pharmacieplus_to_bdd_i2.rb
$: << "./lib"

require 'lib/conversion/propharma_bdd'
require 'lib/conversion/pharmacieplus_bdd'
require 'lib/conversion/bdd_csv'
require 'lib/conversion/bdd_i2'

# PharmaciePlusBdd (xml input file) -> BddI2 (formatted txt file)
reader = XmlConv::Conversion::PharmaciePlusBdd
input  = reader.parse(File.read('input.xml'))
model  = reader.convert(input)
writer = XmlConv::Conversion::BddI2
output = writer.convert(model)
print output.to_s

Run

masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ruby -I lib convert_test_pharmacieplus_bdd_i2.rb

Result

201:CU
202:7601001315747
201:BY
202:1075
201:DP
202:7601001315747
220:xxxx
221:xxxx
222:xxxx
223:xxxx
225:6593
231:secretariat@pharmacieplus.ch
237:61
238:1
300:4
301:20110831
...

Note

  • There is not '251' entry
  • Reader: PharmaciePlusBdd: 'lib/conversion/pharmacieplus_bdd'
  • Writer: BddI2: 'lib/conversion/bdd_i2'

Note

  • 251 entry is described in lib/xmlconv/i2/order.rb
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ grep -r 251 lib/
lib/xmlconv/i2/order.rb:          output << sprintf("250:ADE\n251:%s\n", @ade_id)
  • But this library is not required in lib/conversion/bdd_i2.rb
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ head lib/conversion/bdd_i2.rb 
require 'xmlconv/i2/document'
require 'xmlconv/i2/address'
require 'xmlconv/i2/position'

Experiment

  • lib/conversion/bdd_i2.rb
        def _doc_add_delivery(doc, delivery, sender_id='YWESEE')
          order = I2::Order.new
          order.sender_id = sender_id
          # customer_id is in reality the delivery_id assigned by the
          # customer - the slight confusion is due to automatic naming
          #transaction_id = delivery.customer_id
          #order.ade_id = order.delivery_id = transaction_id
          #doc.header.transaction_id = sprintf(transaction_id.to_s.rjust(8, '0'))
          transaction_id = delivery.customer_id
          order.ade_id = order.delivery_id = transaction_id
          doc.header.transaction_id = sprintf(transaction_id.to_s.rjust(8, '0'))

Run

masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ruby convert_test_pharmacieplus_to_bdd_i2.rb

Result

 001:7601001000681
 002:ORDERX
 003:220
 010:KD7601001315747_00005286.dat
 100:YWESEEPP
 101:5286
 201:CU
 202:7601001315747
 201:BY
 202:1075
 201:DP
 202:7601001315747
 220:xxxx
 221:xxxx
 222:xxxx
 223:xxxx
 225:6593
 231:xxxx
 237:61 
 238:1
 250:ADE
 251:5286
 300:4
 301:20110831

Note

Check it on ftp server

Run

  • masa@masa ~/ywesee/bbmb.ch $ bin/bbmbd
  • masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ bin/xmlconvd config="etc/xmlconv.yml"

Setup

  • copy input.xml to polling directory, "/home/masa/ywesee/globopharm.xmlconv.bbmb.ch/xml_orders"
  • wait one minute
  • check ftp server and xmlconv server (http://masapharm.xmlconv.bbmb.ch/)
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ls xml_orders/
input.xml
...
(one minute later)
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ ls xml_orders/
masa@masa ~/ywesee/globopharm.xmlconv.bbmb.ch $ 

Result

  • '251' entry is shown on xmlconv server but xmlconv server says 'FTP Fehler'

Experiment

  • lib/xmlconv/util/application.rb
      def _execute(transaction)
p "getin _execute"
        transaction.transaction_id = next_transaction_id
p "B"
        transaction.execute
p "C"
        #transaction.postprocess
        ret = transaction.postprocess
p "D"
ret
      rescue Exception => error
  • Create '/IN/Temp' directory on the FTP server

Result

"getin poll_sources"
"getin load_sources"
CONFIG.polling_file = "/home/masa/ywesee/globopharm.xmlconv.bbmb.ch/etc/polling.yml"
source.class = XmlConv::Util::PollingMission
transaction.class = XmlConv::Util::Transaction
"getin _execute"
"B"
"C"
...

Result

  • The result files are shown on the FTP server

IMPORTANT

  • The postprocess needs '/IN/Temp' directory on the FTP server
  • This setting is written in etc/polling.yml file
tmp_destination: "/IN/Temp"

Debug miss-matching between file name and 010 entry

Email

z.B.

masa@masa ~/work $ grep -r KD7601001005631_00005286 KD7601001005631_00005286012.dat 
010:KD7601001005631_00005286.dat

Note

  • In fact, a file name have 3 digits more are added to the 010 entry

suspend

Debug 'FTP Fehler'

Problem

  • FTP transaction succeeds but the status on xmlconv server becomes 'Fehler'

Experiment

  • (Run/Stop) bin/bbmbd
  • Run bin/xmlconvd

Result

  • The status becomes 'ftp_ok' if bin/bbmbd is running at the same time
  • But the status becomes 'Fehler' if bin/bbmbd is NOT running at the same time

Note

  • If the 3 lines are commented out above, the transaction_id (suffix of file name) becomes 'date+time+index' as I updated on 27th Jul.
  • If they are commented in again, the transaction_id becomes the old format

Experiment

  • lib/conversion/bdd_i2.rb
        def _doc_add_delivery(doc, delivery, sender_id='YWESEE')
          order = I2::Order.new
          order.sender_id = sender_id
          # customer_id is in reality the delivery_id assigned by the
          # customer - the slight confusion is due to automatic naming
          transaction_id = delivery.customer_id
          order.ade_id = order.delivery_id = transaction_id
          #doc.header.transaction_id = sprintf(transaction_id.to_s.rjust(8, '0'))
          order.add_date(I2::Date.from_date(::Date.today, :order, :order))

Result

  • file name becomes with date+time+index, and 251 entry is also shown

Note

  • ade_id is necessary to show 251 entry, but doc.header.transaction_id is not necessary to make a suffix of file name like date+time+index

Commit

Updated migel search

Problem

  • At the moment, the search function searchs only one of Group, Subgroup or Migelid
  • It should search all of them at them same time and show all the results

Note

  • src/util/oddbapp.rb#search_migel_products should be updated
    def search_migel_products(query, lang)
      migel_code = if query =~ /(\d){9}/
                     query.split(/(\d\d)/).select{|x| !x.empty?}.join('.')
                   elsif query =~ /(\d\d\.){4}\d/
                     query
                   end
      if migel_code
         MIGEL_SERVER.migelid.search_by_migel_code(migel_code)
      else
        search_method = 'search_by_name_' + lang.downcase.to_s
        MIGEL_SERVER.search_migelid_by_name(query, lang.downcase.to_s)
      end
    end
  • lib/migel/util/server.rb#search_migel_by_name
      def search_migelid_by_name(query, lang)
        search_method = 'search_by_name_' + lang
        result = []

        if groups = Migel::Model::Group.send(search_method, query) and !groups.empty?
          groups.each do |group|
            result.concat group.subgroups.collect{|sg| sg.migelids}.flatten
          end
        end
        if subgroups = Migel::Model::Subgroup.send(search_method, query) and !subgroups.empty?
          result.concat subgroups.collect{|sg| sg.migelids}.flatten
        end
        result.concat Migel::Model::Migelid.send(search_method, query)
        ODBA::DRbWrapper.new(result.uniq)
      end

Run

  • bin/oddbd
  • bin/currencyd
  • bin/migeld

Access (search 'material')

Result

  • Online
  • Local

Note

  • The reason of the difference above is as follows
    1. Online server searches the query not only names of Group, Subgroup, Migelid but also @description data by using 'migel_fulltext_index_de(fr)' table
    2. Local server searches only the names of Group, Subgoup, and Migelid.

Next

  • Create a fulltext table

Question

  • How to create fulltext table as well as oddb.org
view · edit · sidebar · attach · print · history
Page last modified on March 12, 2013, at 02:07 PM