view · edit · sidebar · attach · print · history

20160725-debug-request-encoding-sandoz-xmlconv-bbmb-ch

<< 20160726-migrate-davaz-com-fix-unit-test-sandoz-xmlconv-bbmb-ch | Index | 20160722-debug-incoming-request-conversion-sandoz-xmlconv-bbmb-ch >>

Summary

  • Debug request encoding on sandoz.xmlconv.bbmb.ch

Commits / Patches / Pull Requests, Scripts

Index

Debug request encoding sandoz.xmlconv.bbmb.ch

sandoz.xmlconv.bbmb.ch might have still strange encoding trouble..
Today, I would like to make sure that encoding (internal/external, in Ruby 1.8.6 world and drb) is fine.

Fix plugin loading issue

At first, I've started to clean boot process, because it is a little bit noisy.
This disturbs debugging of another issues.

I've faced this issue on Unit-Test. 

[1] 2.3.1-p112(XmlConv::Util)> rpath
=> "conversion/wbmb_bdd"
[2] 2.3.1-p112(XmlConv::Util)> require rpath
/path/to/gems/ruby/2.3.0/gems/soap4r-1.5.8/lib/xsd/xmlparser.rb:22: warning: already initialized constant \
XSD::XMLParser::NSParseRegexp
/path/to/gems/ruby/2.3.0/gems/soap4r-1.5.8/lib/xsd/xmlparser.rb:22: warning: previous definition of NSParseRegexp was here
RuntimeError: XML processor module not found.

This patch avoids this loading error.
I moved this patch from unit-test to application. 

module XSD                                                                      
  module XMLParser                                                              
    class << self                                                               
      alias :orig_constants :constants                                          
      def constants                                                             
        orig_constants.map(&:to_s)                                              
      end                                                                       
    end                                                                         
  end                                                                           
end

Commit:Fix xml processor module not found issue

Fix kcode warning from soap4r

D, [2016-07-25T09:18:12.906745 #12305] DEBUG -- XmlConv2: loading plugin: 'conversion/wbmb_bdd' (wbmb_bdd.rb)
/path/to/gems/ruby/2.3.0/gems/soap4r-1.5.8/lib/xsd/charset.rb:13: warning: variable $KCODE is no longer effective

I know already also about this.
Then I've change to no convert encoding in xsd (soap4r) library. (because it uses iconv for old ruby)

Commit:Fix charset encoding issue

Meta tag and http-equiv

Xmlconv does not valid meta tag for utf-8 and does not have http response header for encoding.
I think this causes strange character error on browser (ui).

Then append following meta tag (HTML 4.01, same as <meta charset="UTF-8"> in HTML5) 

<meta content="tex/html;charset=UTF-8" http-equiv="content-type">

And also update http-respones header to provide encoding. 

Content-Type	
text/html;charset=UTF-8

This provides fine characters :)

Before

After

Commits:
* Add charset as utf-8 in HTTP_HEADER and META_TAGS [sandoz.xmlconv.bbmb.ch]

Other strange character issues were also fixed by this.

Debug incoming request encoding

We have still encoding issue:
# this order is recorded twice. this means sandoz.xmlconv.bbmb.ch does not return 200.

This request is ISO-8859-1 encoding. 

# client number and password are masked
<?xml version="1.0" encoding="ISO-8859-1"?>
<customerOrder xmlns:xsd="http://www.w3.org/2001/XMLSchema" ... version="1.0">
	<client number="XXXXXXXXXXX" password="XXXXX" />
	<orderHeader deliveryDate="2016-07-22"
		referenceNumber="220720161" mannerOfTransport="tour"
		phoneCallDesired="false" urgent="false">
		<deliveryAddress line1="Rt Cantonale 90 CP. 16"
			line4="Pharmacie V�troz" line5PostalCode="1963"
			line5City="V�troz">
			<addressLine2And3Text line2="V�troz" line3="V�troz" />
		</deliveryAddress>
	</orderHeader>
	<orderLines>
		<productOrderLine orderQuantity="3"
			roundUpForCondition="true" backLogDesired="true">
			<pharmaCode id="5195126" />
		</productOrderLine>
		<productOrderLine orderQuantity="10"
			roundUpForCondition="true" backLogDesired="true">
			<pharmaCode id="2323960" />
		</productOrderLine>
		<productOrderLine orderQuantity="2"
			roundUpForCondition="true" backLogDesired="true">
			<pharmaCode id="5006763" />
		</productOrderLine>
		<productOrderLine orderQuantity="2"
			roundUpForCondition="true" backLogDesired="true">
			<pharmaCode id="5356500" />
		</productOrderLine>
	</orderLines>
</customerOrder>

Note

How to change and Open with encoding with vim.
I do it like this:

# change encoding
:set fileencoding=latin1
:w

# re:open with encoding
:e ++enc=latin1

Return (api) response with charset in header

API Response body has still strange character. Encoding is already UTF-8.
I think this is caused also because api response header has not valid charset. 

% curl -H "Content-Type: text/xml" http://sandoz.xmlconv.bbmb.ch.on.klewenalp/propharma -d @tool/double.xml
<?xml version='1.0' encoding='UTF-8'?>
<customerOrderResponse backLogDesired='false' ...>
<clientResponse number='xxxxxxxxx'/>
  <orderHeaderResponse referenceNumber='220720161'>
    <deliveryAddress line1='Rt Cantonale 90 CP. 16' line4='Pharmacie Vétroz' line5City='Vétroz' line5PostalCode='1963'>
    ...
    <addressLine2And3Text line2='Vétroz' line3='Vétroz'/>
    </deliveryAddress>
    ...
</customerOrderResponse>

Commit: Fix incoming request encoding issue [commit, xmlconv]

Fix Encoding in XmlParser sandoz.xmlconv.bbmb.ch

There 2 points to fix this:

  • API does not return valid charset (UTF-8) in response header. (body is already utf-8, some clients might confuse this)
  • Remove unnecessary encoding conversion in sandoz.xmlconv.bbmb.ch (I've changed it in xmlconv gem)

Commits:
* Create base xml-parser

Note about (current) production setup

On production, some strange characters are still there.
But its are already fixed.

I think this is caused by load path issue.

RubyAddPath (in Apache config) adds just path at the end of $LOAD_PATH.
This does not mean that application use specified version.
Currently, production server has many applications against same gem environment.
So we cannot uninstall easily old version from this gem environment.

If environment has other version gem in path, other version might be used.
So, current these issue on production needs dependency management by Bundler.
In addition,
We have 2 rubies for application.

  • mod_ruby use ruby 1.8.6
  • mod_ruby run *.rbx via Apache and search gems from gem186 environment.
  • Apache configuration has many RubyAddPath to control gem dependencies.
  • Request is passed to Ruby 2.3.1 world (via DRb), and this ruby has another load path. 
This just append path at the End.
RubyAddPath /path/to/ruby/gems/ruby/2.3.0/gems/sbsm-1.2.3/lib

This avoids unnecessary these setup issues... 

# setup (install gems only for this application)
$ bundle install --path /path/to/somewhere

# run script (z.B. in daemontools)
$ bundle exec xmlconvd \                                                          
  config=/var/www/sandoz.xmlconv.bbmb.ch/etc/config.yml

Maybe next time.

view · edit · sidebar · attach · print · history
Page last modified on July 25, 2016, at 07:20 PM