view · edit · sidebar · attach · print · history

< Niklaus.20170626-oddb-org-rack | Index | Niklaus.20170620-sandoz-xmlconv >>

20170620-sandoz-xmlconv

Summary

  • Port oddb.org to use rack
  • Fix submitting orders to sandoz.xmlconv
  • Keep in Mind

Commits

Index

Fix submitting orders to sandoz.xmlconv (Fiddler tests)

Got yesterday the following error

NoMethodError at /de/sbsm
undefined method 'deliveries' for nil:NilClass
transction.model.deliveries:each_with_index do | delivery, idx|

and while handling a transaction

I, [2017-06-20T15:32:45.197687 #27720]  INFO -- : application.rb:44:in `_execute' Starting_transaction 33901
I, [2017-06-20T15:32:45.197898 #27720]  INFO -- : application.rb:51:in `rescue in _execute' error_transaction 33901 error "\xC3" from ASCII-8BIT to UTF-8 
["/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.6/lib/xmlconv/util/transaction.rb:32:in `encode'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.6/lib/xmlconv/util/transaction.rb:32:in `execute'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.6/lib/xmlconv/util/application.rb:45:in `_execute'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.6/lib/xmlconv/util/application.rb:36:in `execute'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.6/lib/xmlconv/util/application.rb:148:in `execute_with_response'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.6/lib/xmlconv/state/login.rb:30:in `initialize'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/sbsm-1.4.5/lib/sbsm/session.rb:411:in `new'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/sbsm-1.4.5/lib/sbsm/session.rb:411:in `logout'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/sbsm-1.4.5/lib/sbsm/session.rb:251:in `block in process_rack'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/sbsm-1.4.5/lib/sbsm/session.rb:235:in `synchronize'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/sbsm-1.4.5/lib/sbsm/session.rb:235:in `process_rack'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/sbsm-1.4.5/lib/sbsm/app.rb:119:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/static.rb:149:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/common_logger.rb:33:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/tempfile_reaper.rb:15:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:49:in `_call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/lint.rb:37:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/show_exceptions.rb:23:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/common_logger.rb:33:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/chunked.rb:54:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/content_length.rb:15:in `call'",
"/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/rack-2.0.3/lib/rack/handler/webrick.rb:86:in `service'",
"/usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:140:in `service'",
"/usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/httpserver.rb:96:in `run'",
"/usr/local/ruby-2.4.0/lib/ruby/2.4.0/webrick/server.rb:290:in `block in start_thread'"]

Submitting the same XML to oddb-ci2 worked without a hitch. Why?

On fastpower we as language enviornments LC_COLLATE=POSIX and LANG=en_US.UTF-8. On oddb-ci2 we only have LANG=en_US.UTF-8.

Ran sandox.xmlconv with LC_COLLATE=POSIX and still got not error. Using pry

[2] pry(#<XmlConv::Util::Transaction>)> ENV['LC_COLLATE']
=> "POSIX"
[3] pry(#<XmlConv::Util::Transaction>)> @input.valid_encoding?
=> true
[5] pry(#<XmlConv::Util::Transaction>)> @input.encoding
=> #<Encoding:UTF-8>
[6] pry(#<XmlConv::Util::Transaction>)> puts @input
<?xml version "1.0" encoding="utf-8"?><customerOrder xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" compressionDesired="false" productDescriptionDesired="true" backLogDesired="true" language="fr" roundUpForCondition="true" version="1.0">  <client number="4100602840" password="MDPOK"/>  <orderHeader deliveryDate="2017-06-20" referenceNumber="200620171" mannerOfTransport="tour" phoneCallDesired="false" urgent="false">    <deliveryAddress line1="16, bd des Philosophes" line4="Pharmacie des PHILOSOPHES" line5PostalCode="1205" line5City="Genève">      <addressLine2And3Text line2="Genève" line3="des PHILOSOPHES"/>    </deliveryAddress>  </orderHeader>  <orderLines>    <productOrderLine orderQuantity="2" roundUpForCondition="true" backLogDesired="true">      <pharmaCode id="3023007"/>    </productOrderLine>    <productOrderLine orderQuantity="2" roundUpForCondition="true" backLogDesired="true">      <pharmaCode id="1586108"/>    </productOrderLine>    <productOrderLine orderQuantity="1" roundUpForCondition="true" backLogDesired="true">      <pharmaCode id="3463410"/>    </productOrderLine>    <productOrderLine orderQuantity="1" roundUpForCondition="true" backLogDesired="true">      <pharmaCode id="5621416"/>    </productOrderLine>    <productOrderLine orderQuantity="1" roundUpForCondition="true" backLogDesired="true">      <pharmaCode id="4714216"/>    </productOrderLine>  </orderLines></customerOrder>

Trying with test/data/umlaut_utf_8_problem.xml and LC_COLLATE=posix did not produce any error. Looking at our unit tests. We have one unit test named test_execute. Creating a new one test_execute_utf_8_problem. Was able to create a test case important steps were

  • Load test file in binary mode File.open("#{test_data_dir}/umlaut_iso8859.xml", 'rb').read
  • Convert UTF-8 to ISO-8859-1 using iconv -f utf8 -t iso8859-14 test/data/umlaut_utf_8_problem.xml -o test/data/umlaut_iso8859.xml

Now the test fails like follows

  1) Error:
XmlConv::Util::TestTransaction#test_execute_utf_8_problem:
Encoding::UndefinedConversionError: "\xEA" from ASCII-8BIT to UTF-8
    /home/niklaus/git/xmlconv/lib/xmlconv/util/transaction.rb:36:in `encode'
    /home/niklaus/git/xmlconv/lib/xmlconv/util/transaction.rb:36:in `execute'
    test/test_util/transaction.rb:126:in `test_execute_utf_8_problem'

And Zeno was right using @input.force_encoding("UTF-8") fixed the problem, but only if we encode it afterwards a second time.

Pushed the following commits

Release xmlconv 1.1.7 and pushed commit Use xmlconv 1.1.7

Installed and started it on fastpower. Tested it with confuse.xml and umlaut_ . Okay. Also already got another command where we have in the log/2017/06/21/app.logfile an entry

D, [2017-06-21T12:04:53.388517 #31495] DEBUG -- : login.rb:17:in `initialize'  xml_src ASCII-8BIT now <?xml version="1.0" encoding="utf-8"?><customerOrder xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" compressionDesired="false" productDescriptionDesired="true" backLogDesired="true" language="fr" roundUpForCondition="true" version="1.0">

Remarked that my curl test pushed it as UTF-8 as seen in the log

D, [2017-06-21T12:04:50.042920 #31495] DEBUG -- : login.rb:17:in `initialize'  xml_src UTF-8 now <?xml version "1.0" encoding "utf-8"?><customerOrder xmlns:xsd="http://www.w3.org/2001/XMLSchema"     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"     compressionDesired="false" productDescriptionDesired="true"     backLogDesired="true" language="fr"     roundUpForCondition="true" version="1.0">  <client number="99" password="" />  <orderHeader deliveryDate="2017-06-06"       referenceNumber="060620172" mannerOfTransport="tour"       phoneCallDesired="false" urgent="false">    <deliveryAddress line1="136, Ch. De-La-Montagne"         line4="Pharmacie GOUDA" line5PostalCode="1224"         line5City="Ch&#65533;ne-Bougeries">      <addressLine2And3Text line2="Ch&#65533;ne-Bougeries" line3="GOUDA"           />    </deliveryAddress>  </orderHeader>  <orderLines>    <productOrderLine orderQuantity="10"         roundUpForCondition="true" backLogDesired="true">      <pharmaCode id="3463433" />    </productOrderLine>    <productOrderLine orderQuantity="2"         roundUpForCondition="true" backLogDesired="true">      <pharmaCode id="3029702" />    </productOrderLine>  </orderLines></customerOrder>

Trying to add --raw option and the --data-binary, eg. curl -v http://sandoz.xmlconv.bbmb.ch/propharma --data-binary @test/data/umlaut_iso8859.xml. But this failed, too. The uploaded content as seen by sandoz.xmlconv.bbmb.ch was always UTF-8.

Port oddb.org to use rack

Export jobs to check are:

  • jobs/export_oddb2_csv (okay)
  • jobs/export_oddb2tdat (okay)
  • jobs/export_oddb_csv (okay)
  • jobs/export_swissdrug-update (okay)
  • jobs/export_teilbarkeit (okay)
  • jobs/export_daily (okay)

Running all watirs tests produced the following resumen

Finished in 53 minutes 17 seconds (files took 24.29 seconds to load)
179 examples, 59 failures, 34 pending

Failed examples:

rspec ./spec/rezept_and_instantsearch_spec.rb:571 # ch.oddb.org should be possible to print a presciption with 10 drugs
rspec ./spec/rezept_and_instantsearch_spec.rb:250 # ch.oddb.org should be possible to add drugs after delete_all when neither ZSR nor comment given
rspec ./spec/rezept_and_instantsearch_spec.rb:452 # ch.oddb.org should not loose existing comment after adding a new prescription
rspec ./spec/rezept_and_instantsearch_spec.rb:288 # ch.oddb.org should be possible to add drugs after delete_all when no ZSR is given but a comment
rspec ./spec/searchbar_spec.rb[1:2] # ch.oddb.org should be possible to find 1,25-Dihydroxycholecalciferol when searching via 125 in analysen
rspec ./spec/searchbar_spec.rb:116 # ch.oddb.org should should be possible to add and delete several drugs:
rspec ./spec/searchbar_spec.rb:339 # ch.oddb.org should show no drugs for Fortex via unwanted effects search
rspec ./spec/searchbar_spec.rb:135 # ch.oddb.org should should be possible to delete all drugs:
rspec ./spec/searchbar_spec.rb[1:5] # ch.oddb.org should should work (58868 Actemra) with Dos./Anw. and Kinder
rspec ./spec/searchbar_spec.rb:391 # ch.oddb.org should set best_result when searching Rivoleve via search_type
rspec ./spec/searchbar_spec.rb[1:6] # ch.oddb.org should should work (58868 Actemra) with Interakt. and Tocilizumab
rspec ./spec/searchbar_spec.rb[1:4] # ch.oddb.org should should work (58868 Actemra) with Unerw.Wirkungen and Kopfschmerzen
rspec ./spec/admin_spec.rb[1:1] # ch.oddb.org should be possible to upload dummy_patinfo.pdf to a given package
rspec ./spec/admin_spec.rb[1:2] # ch.oddb.org should be possible to upload dummy_patinfo_2.pdf to a given package
rspec ./spec/admin_spec.rb:129 # ch.oddb.org should be possible to create a CompanyUser
rspec ./spec/evidentia_spec.rb:49 # ch.oddb.org should list C09DB02 before C09DX03 when looking for Sevikar
rspec ./spec/evidentia_spec.rb:172 # ch.oddb.org should list trademark first e.g. Duodopa
rspec ./spec/evidentia_spec.rb:165 # ch.oddb.org should display lamivudin with SO and SG in category (price comparision)
rspec ./spec/evidentia_spec.rb:60 # ch.oddb.org should list Keppra at the top when searching for Levetiracetam
rspec ./spec/evidentia_spec.rb:143 # ch.oddb.org should contain a link to the fachinfo for Lamivudin-Zidovudin
rspec ./spec/evidentia_spec.rb:72 # ch.oddb.org should list all SL products before the Non-SL
rspec ./spec/evidentia_spec.rb:127 # ch.oddb.org should contain a link to the FI for the drug when in price comparison
rspec ./spec/evidentia_spec.rb:117 # ch.oddb.org should contain a link to the price comparision in price public
rspec ./spec/evidentia_spec.rb:218 # ch.oddb.org should list Levetiracetam Desitin with a link to the product overview
rspec ./spec/evidentia_spec.rb:66 # ch.oddb.org should list Levetiracetam Desitin at the top when searching for Levetiracetam Desitin
rspec ./spec/evidentia_spec.rb:93 # ch.oddb.org should not contain a column Fachinfo
rspec ./spec/evidentia_spec.rb:106 # ch.oddb.org should contain a link to the limiation in Sevikar HCT preparation
rspec ./spec/evidentia_spec.rb:210 # ch.oddb.org should display Cellcept before other
rspec ./spec/evidentia_spec.rb:151 # ch.oddb.org should display a limitation link for Sevikar HCT
rspec ./spec/download_spec.rb:73 # ch.oddb.org should be possible to run grant_download oddb2.csv
rspec ./spec/download_spec.rb:27 # ch.oddb.org should download the results of a search to Marcoumar
rspec ./spec/download_spec.rb:45 # ch.oddb.org should be possible to run a bin/admin command
rspec ./spec/paypal_spec.rb:86 # ch.oddb.org should be possible to checkout oddb.csv via paypal
rspec ./spec/paypal_spec.rb:177 # ch.oddb.org should be possible to cancel a paypal before login
rspec ./spec/paypal_spec.rb:113 # ch.oddb.org should be checkout via paypal a poweruser
rspec ./spec/paypal_spec.rb:188 # ch.oddb.org should be possible to cancel a paypal after login but before paying
rspec ./spec/paypal_spec.rb:163 # ch.oddb.org should not download a CSV file if the payment was not accepted
rspec ./spec/paypal_spec.rb:140 # ch.oddb.org should return a correct link to a CSV file if the payment is okay
rspec ./spec/interactions_spec.rb:276 # ch.oddb.org after delete all drugs a new search must be possible
rspec ./spec/snapback_spec.rb[1:6] # ch.oddb.org snapback should a working Home button for step 4
rspec ./spec/snapback_spec.rb[1:5] # ch.oddb.org snapback should a working Home button for step 3
rspec ./spec/snapback_spec.rb:107 # ch.oddb.org snapback should allow going back, then forward
rspec ./spec/snapback_spec.rb[1:4] # ch.oddb.org snapback should a working Home button for step 2
rspec ./spec/snapback_spec.rb:174 # ch.oddb.org snapback should work follow correctly the expected paths
rspec ./spec/address_correction_spec.rb:84 # ch.oddb.org should be possible to correct an address for a company
rspec ./spec/smoketest_spec.rb:279 # ch.oddb.org should show correct Tramal Tropfen Lösung zum Einnehmen mit Dosierpumpe (4788/01/035)
rspec ./spec/smoketest_spec.rb:354 # ch.oddb.org should download the results of a search
rspec ./spec/smoketest_spec.rb[1:31] # ch.oddb.org should have a working instant search for Augmentin and takeskosten
rspec ./spec/smoketest_spec.rb:230 # ch.oddb.org should have a link to the english language versions
rspec ./spec/smoketest_spec.rb:127 # ch.oddb.org should display lamivudin with SO and SG in category (price comparision)
rspec ./spec/smoketest_spec.rb:107 # ch.oddb.org should display a limitation link for Sevikar
rspec ./spec/smoketest_spec.rb:397 # ch.oddb.org should display the correct calculation for Bicalutamid Actavis
rspec ./spec/rss_spec.rb[1:1] # ch.oddb.org should have a working RSS-feed hpc
rspec ./spec/rss_spec.rb[1:3] # ch.oddb.org should have a working RSS-feed price_rise
rspec ./spec/rss_spec.rb[1:5] # ch.oddb.org should have a working RSS-feed sl_introduction
rspec ./spec/rss_spec.rb[1:4] # ch.oddb.org should have a working RSS-feed recall
rspec ./spec/rss_spec.rb[1:2] # ch.oddb.org should have a working RSS-feed price_cut
rspec ./spec/rss_spec.rb[1:6] # ch.oddb.org should have a working RSS-feed fachinfo
rspec ./spec/rss_spec.rb:67 # ch.oddb.org should have a working fachinfo-2008

Made some changes to handle the different flavor as I cannot use the approach from mod_ruby where we had an environment flag SetEnv DEFAULT_FLAVOR gcc in the apache conf. But it does not work correctly for oddb-ci2.dyndns.org. Fixed by not converting nil values to string.

Now the flavors work where they a key in CssTemplate and this key is the start of the servername, eg. santesuisse, evidentia. But must fix it for other cases like homeopathy, phyto-pharma, oekk.

Pushed the commits

Remarked that when running the test oddbapp*.rb the log_size thread is started many times. Added a guard to ensure, that it is started only once. Now the unit test pass a lot faster.

Fixing some other failing tests in oddbapp_2.rb. Changed sbsm/session_store.rb to handle clear method.

Pushed commit

view · edit · sidebar · attach · print · history
Page last modified on June 21, 2017, at 05:07 PM