view · edit · sidebar · attach · print · history

< Niklaus.20170620-sandoz-xmlconv | Index | Niklaus.20170614-oddb-org-rack >>

20170619-sandoz-xmlconv

Summary

  • Port oddb.org to use rack
  • Fix submitting orders to sandoz.xmlconv
  • Keep in Mind

Commits

Index

Fix submitting orders to sandoz.xmlconv (Fiddler tests)

We want to use fiddler to submit orders to sandoz.xmlconv. Must use a Windows Virtualbox machine for it. Installing from http://www.fiddler2.com/fiddler2/.

Modified C:\windows\sytem32\drivers\etc\hosts to add the IP of my used host sandoz.xmlconv.bbmb.ngiger.ch. Added confuse.xml. The order was placed correctly. But adding a breakpoint in lib/sbsm/session.rb when processing a POST request to see, how the POST looked liked.

From: /home/niklaus/git/sbsm/lib/sbsm/session.rb @ line 215 SBSM::Session#process_rack:

    210:       @request_path ||= rack_request.path
    211:       @rack_request = rack_request
    212:       @post_content = nil
    213:       if rack_request.request_method.eql?('POST')
    214:         binding.pry
 => 215:          rack_request.params.each do |k, v|
    216:            # needed to test POST requests generated by curl (first parameter) or ARC (second parameter)
    217:           if /xml/i.match(k)
    218:             @post_content = "#{k} #{v}"
    219:             break
    220:           end

[1] pry(#<XmlConv::Util::Session>)> rack_request.params
=> {"<?xml version"=>
  "\"1.0\" encoding=\"utf-8\"?>\n" +
  "<customerOrder xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\" version=\"1.0\" language=\"fr\" productDescriptionDesired=\"true\" communicationSoftwareId=\"ywesee Test\" xmlns=\"http://www.e-galexis.com/schemas/\">\n" +
  "  <client number=\"99\" password=\"\" />\n" +
  "  <orderHeader deliveryDate=\"2016-07-15\" referenceNumber=\"99\"/>\n" +
  "  <orderLines>\n" +
  "    <productOrderLine orderQuantity=\"2\">\n" +
  "      <pharmaCode id=\"2508375\" />\n" +
  "    </productOrderLine>\n" +
  "    <productOrderLine orderQuantity=\"2\">\n" +
  "      <pharmaCode id=\"5195126\" />\n" +
  "    </productOrderLine>\n" +
  "  </orderLines>\n" +
  "</customerOrder>\n"}

Will test with the other xml files, too. Using

  • test/data/umlaut_utf_8_problem.xml
  • test/data/data/gard.xml (responds with expected RuntimeError can't commit empty order)
  • test/data/data/vetroz.xml

Pushed commit Use SBSM 1.4.5

Updating on thinpower as documented under http://dev.ywesee.com/Niklaus/20170613-oddb-org-rack#update-sandoz. This seems to work. Placed an order via confuse.xml which appears as 33855 http://80.218.53.88 19.06.2017 12:08:51 file:/.../var/output/99_20170619120851065.txt Bestellung via BBMB erfolgreich

Uncommenting the smtp_fields in //var/www/sandoz.xmlconv.bbmb.ch/etc/xmlconv.xml to avoid XmlConv2: Net::SMTPAuthenticationError in /var/www/sandoz.xmlconv.bbmb.ch/log/2017/06/19/app.log when polling sources.

Got another empty order 33856 http://194.158.242.196 19.06.2017 13:36:39 file:/.../sandoz.xmlconv.bbmb.ch/var/output Looking at the debug in /var/www/sandoz.xmlconv.bbmb.ch/log/2017/06/19/app.log I see

D, [2017-06-19T13:36:38.802475 #20954] DEBUG -- : app.rb:118:in `call' starting session_id 1d9cd1a2446528e624caa4b789a  session XmlConv::Util::Session /pharmatic: cookies  are {} @cgi NilClass
D, [2017-06-19T13:36:38.802616 #20954] DEBUG -- : session.rb:227:in `process_rack' @post_content is <?xml version="1.0" encoding="utf-8"?><customerOrder xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" version="1.0" language="de" productDescriptionDesired="true" communicationSoftwareId="Pharmatic NETREQ.EXE V1.20.0.4 (Requestdate: 19.06.2017 13:38)" xmlns="http://www.e-galexis.com/schemas/"><client number="4100603668" password="7601001007307" /><orderHeader deliveryDate="2017-06-19" referenceNumber="173551" /><orderLines><productOrderLine orderQuantity="10"><pharmaCode id="2347937" /></productOrderLine><productOrderLine orderQuantity="2"><pharmaCode id="2324072" /></productOrderLine><productOrderLine orderQuantity="10"><pharmaCode id="5797463" /></productOrderLine></orderLines></customerOrder>
D, [2017-06-19T13:36:41.611455 #20954] DEBUG -- : session.rb:412:in `logout' logout "/pharmatic" setting @state 70116712063980 XmlConv::State::Login remember nil XmlConv::Util::KnownUser
D, [2017-06-19T13:36:41.612582 #20954] DEBUG -- : session.rb:256:in `block in process_rack' active_state.trigger state 70116712063980 XmlConv::State::Login remember nil
D, [2017-06-19T13:36:41.612651 #20954] DEBUG -- : session.rb:265:in `block in process_rack' Changing from 70116712063980 to state XmlConv::State::Login 70116712063980 remember nil  XmlConv::Util::KnownUser
D, [2017-06-19T13:36:41.614907 #20954] DEBUG -- : app.rb:150:in `call' finish session_id.1 1d9cd1a2446528e624caa4b789a: matches response.headers['Set-Cookie']
194.158.242.196 - - [19/Jun/2017:13:36:41 +0200] "POST /pharmatic HTTP/1.1" 200 404 2.8132
I, [2017-06-19T13:36:43.522770 #20954]  INFO -- : session.rb:114:in `initialize' initialize th #<SBSM::TransHandler:0x007f8aa319f930> validator XmlConv::Util::Validator app XmlConvApp multi_threaded false
D, [2017-06-19T13:36:43.522934 #20954] DEBUG -- : session.rb:135:in `initialize' session initialized #<XmlConv::Util::Session:0x007f8aab189768> with @cgi #<CGI:0x007f8aab183d18> multi_threaded false app 70116709207600 and user SBSM::UnknownUser @unknown_user XmlConv::Util::KnownUser
D, [2017-06-19T13:36:43.523002 #20954] DEBUG -- : app.rb:118:in `call' starting session_id 1fbfb6258628fb96a08c0159ead  session XmlConv::Util::Session /: cookies  are {} @cgi NilClass
D, [2017-06-19T13:36:43.523350 #20954] DEBUG -- : session.rb:412:in `logout' logout "/" setting @state 70116776320380 XmlConv::State::Login remember nil XmlConv::Util::KnownUser
D, [2017-06-19T13:36:43.524581 #20954] DEBUG -- : session.rb:256:in `block in process_rack' active_state.trigger state 70116776320380 XmlConv::State::Login remember nil
D, [2017-06-19T13:36:43.524651 #20954] DEBUG -- : session.rb:265:in `block in process_rack' Changing from 70116776320380 to state XmlConv::State::Login 70116776320380 remember nil  XmlConv::Util::KnownUser
D, [2017-06-19T13:36:43.524739 #20954] DEBUG -- : session.rb:504:in `set_cookie_input' cookie_set_or_get language de

Uncommented all smtp_fields in /var/www/sandoz.xmlconv.bbmb.ch/etc/xmlconf.yml and restored as documented yesteray. Submitting adler.xml locally and on fastpower was okay.

My local setup does not have a polling_file declaration in etc/xmlconv.etc. I think that I have therefore never produced the error

E, [2017-06-19T12:45:10.887812 #16745] ERROR -- XmlConv2: Net::SMTPAuthenticationError
535-5.7.8 Username and Password not accepted. Learn more at

/usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/smtp.rb:981:in `check_auth_response'
/usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/smtp.rb:736:in `auth_plain'
/usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/smtp.rb:728:in `authenticate'
/usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/smtp.rb:565:in `do_start'
/usr/local/ruby-2.4.0/lib/ruby/2.4.0/net/smtp.rb:518:in `start'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/mail-2.6.6/lib/mail/network/delivery_methods/smtp.rb:111:in `deliver!'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/mail-2.6.6/lib/mail/message.rb:2149:in `do_delivery'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/mail-2.6.6/lib/mail/message.rb:239:in `deliver'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/mail-2.6.6/lib/mail/mail.rb:141:in `deliver'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/mail.rb:27:in `notify'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/polling_manager.rb:197:in `rescue in block in poll_sources'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/polling_manager.rb:186:in `block in poll_sources'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/polling_manager.rb:179:in `block in load_sources'
/usr/local/ruby-2.4.0/lib/ruby/2.4.0/psych/deprecated.rb:32:in `each'
/usr/local/ruby-2.4.0/lib/ruby/2.4.0/psych/deprecated.rb:32:in `load_documents'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/polling_manager.rb:178:in `load_sources'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/polling_manager.rb:185:in `poll_sources'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/application.rb:172:in `block (2 levels) in start_polling'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/application.rb:170:in `loop'
/var/www/sandoz.xmlconv.bbmb.ch/vendor/ruby/2.4.0/gems/xmlconv-1.1.5/lib/xmlconv/util/application.rb:170:in `block in start_polling'

on oddb-ci2. Readding the polling_file on oddb_ci2.

On fastpower sandoz.xmlconf still uses via /usr/local/bin/xmlconv the version 1.0.1 of xmlconv. Therefore we see in /usr/local/bin/xmlconvd lines like

Skipping as RUBY_VERSON is 2.3.1
XmlConv::Util::Mail.notify xmlconv@ywesee.com -> ["ngiger@ywesee.com"] subject: XmlConv2 - Polling-Error

Patching /usr/local/lib/ruby/gems/2.3.0/gems/xmlconv-1.1.0/lib/xmlconv/util/polling_manager.rb to get some more info about the config.

Created on fastpower two variants of etc/xmlconv.yml, namely etc/xmlconv.yml.pre_rack and etc/xmlconv.yml.with_rack. etc/xmlconv.yml.with_rack has commented out the polling_file and does define the smtp-values for sending e-mails.

Re-adding the commands. If the errors persists, added the commit Debug output for polling and transactions

Port oddb.org to use rack

Using the sbsm 1.4.5 gem (locally built). The main page works. Searches for pharmacies like http://oddb-ci2.dyndns.org/de/gcc/search/zone/pharmacies/search_query/Glarus? work fine. Displaying a pharmacie works, too. Search by practitioner in Glarus and displaying works. Interactions work too. Migel search by Krücke worked, too. Search and display for hospital Glarus worked, too. Search and display for trademark owner Sandoz.

Debugging the trademark search for Nolvadex via breakpoint in to_html of lib/sbsm/state.rb

[1] pry(#<ODDB::Session>)> @state.class
=> ODDB::State::Drugs::Result
[2] pry(#<ODDB::Session>)> @state.model.class
=> ODDB::State::Drugs::Result
[5] pry(#<ODDB::Session>)> @state.model.package_count
=> 8
[6] pry(#<ODDB::Session>)> @state.model.atc_classes.class
=> Array
[7] pry(#<ODDB::Session>)> @state.model.atc_classes.size
=> 1
[9] pry(#<ODDB::Session>)> @state.model.atc_classes.first.code
=> "L02BA01"
[11] pry(#<ODDB::Session>)> @state.model.atc_classes.first.packages.size
=> 14
[12] pry(#<ODDB::Session>)> @state.model.atc_classes.first.active_packages.size
=> 8
[13] pry(#<ODDB::Session>)> @state.model.atc_classes.first.active_packages.first.name
=> "Tamec 20 mg, Filmtabletten"
[14] pry(#<ODDB::Session>)> @state.model.atc_classes.first.active_packages.first.iksnr
=> "54989"
[15] pry(#<ODDB::Session>)> @state.model.atc_classes.first.active_packages.collect{|x| x.iksnr}
=> ["54989", "54989", "49059", "49059", "49059", "49059", "39053", "39053"]
# it hangs while evaluating
# lib/sbsm/state.rb @ line 173 SBSM::State#to_html:
#    172:                def to_html(context)
# => 173:       name = view

Tracking down to super call of init in src/view/resulttemplate.rb -> src/view/publictemplate.rb -> lib/htmlgrid/composite.rb -> lib/htmlgrid/composite.rb (compose)

The warning NoMethodErrror undefined method `size' for nil:NilClass comes from lib/htmlgrid/composite.rb(around line 209)

Tracked the problem to the fact, that ajax script was not get loaded, as we had set multithreaded to false for SBSM. Now we must fix this problem. After changing multi_threaded: false to multi_threaded: true in oddb.org/src/util/session.rb searching drugs worked. Adding some additional log entries to track down the use of the session_id (cookie).

Here is the grep for session_id of a simple search for Nolvadex

D, [2017-06-19T11:12:22.261897 #11424] DEBUG -- : app.rb:118:in `call' starting session_id 1e764a8a48dd6a7d02bbcf0753e  session ODDB::Session /de/gcc/home/: cookies  are {"oddb-preferences"=>"language=de", "_session_id"=>"1e764a8a48dd6a7d02bbcf0753e", "resultview"=>"", "language"=>"de", "search_form"=>"plus", "__utma"=>"120231382.876330393.1496818562.1497595629.1497855722.8", "__utmb"=>"120231382.55.10.1497855722", "__utmc"=>"120231382", "__utmz"=>"120231382.1496818562.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)"} @cgi NilClass
D, [2017-06-19T11:12:26.806669 #11424] DEBUG -- : app.rb:152:in `call' finish session_id.2 1e764a8a48dd6a7d02bbcf0753e: headers {"Content-Length"=>"20931", "Content-Type"=>"text/html; charset=UTF-8", "Cache-Control"=>"private, no-store, no-cache, must-revalidate, post-check=0, pre-check=0", "Pragma"=>"no-cache", "Expires"=>"Mon, 19 Jun 2017 09:12:14 UTC", "P3P"=>"CP='OTI NID CUR OUR STP ONL UNI PRE'", "Set-Cookie"=>"language=de; path=/\nsearch_form=plus; path=/"}
D, [2017-06-19T11:12:33.156202 #11424] DEBUG -- : app.rb:118:in `call' starting session_id 1e764a8a48dd6a7d02bbcf0753e  session ODDB::Session /de/gcc/search/zone/drugs/search_query/Nolvadex/search_type/st_sequence: cookies  are {"oddb-preferences"=>"language=de", "_session_id"=>"1e764a8a48dd6a7d02bbcf0753e", "resultview"=>"", "language"=>"de", "search_form"=>"plus", "__utmt"=>"1", "__utma"=>"120231382.876330393.1496818562.1497595629.1497855722.8", "__utmb"=>"120231382.56.10.1497855722", "__utmc"=>"120231382", "__utmz"=>"120231382.1496818562.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)"} @cgi NilClass
D, [2017-06-19T11:12:33.214132 #11424] DEBUG -- : app.rb:118:in `call' starting session_id 4179925c644b5446a409fabd21  session ODDB::Session /de/gcc/ajax_swissmedic_cat/reg/39053/seq/02/pack/039: cookies  are {} @cgi NilClass
D, [2017-06-19T11:12:40.675502 #11424] DEBUG -- : app.rb:150:in `call' finish session_id.1 4179925c644b5446a409fabd21: matches response.headers['Set-Cookie']
D, [2017-06-19T11:12:40.690417 #11424] DEBUG -- : app.rb:118:in `call' starting session_id 18922391a433221ad4f4f6f0a90  session ODDB::Session /de/gcc/ajax_swissmedic_cat/reg/39053/seq/02/pack/047: cookies  are {} @cgi NilClass
D, [2017-06-19T11:12:40.703536 #11424] DEBUG -- : app.rb:150:in `call' finish session_id.1 18922391a433221ad4f4f6f0a90: matches response.headers['Set-Cookie']
D, [2017-06-19T11:12:40.711646 #11424] DEBUG -- : app.rb:152:in `call' finish session_id.2 1e764a8a48dd6a7d02bbcf0753e: headers {"Content-Length"=>"33300", "Content-Type"=>"text/html; charset=UTF-8", "Cache-Control"=>"private, no-store, no-cache, must-

The main session 1e764a8a48dd6a7d02bbcf0753e started to sub-session 4179925c644b5446a409fabd21 and 18922391a433221ad4f4f6f0a90. Using http_fox to see whether in the pre-rack we used also three sessions. Here some snippets from the httpfox

(Request-Zeile)	GET / HTTP/1.1
Host	ch.oddb.org
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language	de-CH,en-US;q=0.7,en;q=0.3
Accept-Encoding	gzip, deflate
Cookie	__utma=69243458.1492321151.1496340195.1496340195.1496340195.1; __utmz=69243458.1496340195.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); ["oddb-preferences=language%3Dde

(Request-Zeile)	GET /de/gcc/search/zone/drugs/search_query/Nolvadex/search_type/st_sequence HTTP/1.1
Host	ch.oddb.org
User-Agent	Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language	de-CH,en-US;q=0.7,en;q=0.3
Accept-Encoding	gzip, deflate
Referer	http://ch.oddb.org/
Cookie	__utma=69243458.1492321151.1496340195.1496340195.1497864556.2; __utmz=69243458.1496340195.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); ["oddb-preferences=language%3Dde; _session_id=bf7963d64eec7cef454262a7ebfdfe1b; __utmb=69243458.1.10.1497864556; __utmc=69243458; __utmt=1

But I do not see any request for de/gcc/ajax_swissmedic_cat/reg/39053/seq/02/pack/039. mod_ruby based SBSM fetched this content in a different way. But on thinpower I using grep -w GET /var/www/oddb.org/log/oddb/access_log showed also many entries like 62.12.131.38 - - [19/Jun/2017:00:00:10 +0200] "GET /en/gcc/ajax_swissmedic_cat/reg/58327/seq/02/pack/020 HTTP/1.1" 200 1287 "-" "Ruby"

To search for long lasting requests it would suffice to analyse the last value when grepping for the word GET in the app_log, as there we have the duration (in seconds) of the query, e.g. the (first search) vor Nolvadex took 7.557 seconds. 172.25.1.75 - - [19/Jun/2017:11:12:40 +0200] "GET /de/gcc/search/zone/drugs/search_query/Nolvadex/search_type/st_sequence HTTP/1.1" 200 33300 7.5570 whereas the second one took less and 0.1 seconds (172.25.1.75 - - [19/Jun/2017:11:27:04 +0200] "GET /de/gcc/search/zone/drugs/search_query/Nolvadex/search_type/st_sequence HTTP/1.1" 200 33300 0.0708)

Checking exports. Here the lists of the exports run and the generateded files

  • jobs/export_galenic
    • -rw-r--r-- 1 apache apache 337437 Jun 19 12:52 data/downloads/galenic_forms.yaml
    • -rw-r--r-- 1 apache apache 21889 Jun 19 12:52 data/downloads/galenic_forms.yaml.zip
    • -rw-r--r-- 1 apache apache 21773 Jun 19 12:52 data/downloads/galenic_forms.yaml.gz
    • -rw-r--r-- 1 apache apache 1365 Jun 19 12:52 data/downloads/galenic_groups.yaml.zip
    • -rw-r--r-- 1 apache apache 1247 Jun 19 12:52 data/downloads/galenic_groups.yaml.gz
    • -rw-r--r-- 1 apache apache 6871 Jun 19 12:52 data/downloads/galenic_groups.yaml
  • jobs/export_csv
    • -rw-r--r-- 1 apache apache 6251021 Jun 19 12:59 data/downloads/oddb.csv
    • -rw-r--r-- 1 apache apache 989551 Jun 19 12:59 data/downloads/oddb.csv.gz
    • -rw-r--r-- 1 apache apache 989647 Jun 19 12:59 data/downloads/oddb.csv.zip
    • -rw-r--r-- 1 apache apache 6618163 Jun 19 12:59 data/downloads/oddb2.csv
    • -rw-r--r-- 1 apache apache 1004913 Jun 19 12:59 data/downloads/oddb2.csv.zip
    • -rw-r--r-- 1 apache apache 1004815 Jun 19 12:59 data/downloads/oddb2.csv.gz
  • jobs/export_ddd_csv (Had error sending confirmation email)
    • -rw-r--r-- 1 apache apache 1040204 Jun 19 13:15 ddd.csv
  • jobs/export_fachinfo_chapter Nothing was exported did see does not found any chapter/description
  • jobs/export_flickr_ean (Had error sending confirmation email)
    • -rw-r--r-- 1 apache apache 10544 Jun 19 13:31 data/downloads/flickr_ean_export.csv
    • -rw-r--r-- 1 apache apache 3499 Jun 19 13:31 data/downloads/flickr_ean_export.csv.zip
    • -rw-r--r-- 1 apache apache 3377 Jun 19 13:31 data/downloads/flickr_ean_export.csv.gz
  • jobs/export_oddb2_csv
  • jobs/export_oddb2tdat
  • jobs/export_oddb_csv
  • jobs/export_swissdrug-update
  • jobs/export_teilbarkeit
  • jobs/export_daily

Watir tests cannot login, whereas testing by myself I had no problem logging in. Why? Firefox pops up now "Insecure password warning in Firefox". Can we suppress this?

view · edit · sidebar · attach · print · history
Page last modified on June 19, 2017, at 05:10 PM