view · edit · sidebar · attach · print · history

Index>

20161005-davaz-rack

Summary

  • Trying to port sbsm/davaz.com to use the rack webserver
  • Fix error Duplicate Jobs
  • port sandoz.xmlconv to rack
  • Keep in Mind

Commits

Index

Fix error Duplicate Jobs

Got several times in september error mail as seen by the following lines in log/oddb/debug/2016/09.log 

2016-09-06 07:01:37 CEST Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Duplicate Job: import_daily - 09/2016 size 156 with 0 attachments. Task: Job import_daily
2016-09-10 07:01:36 CEST Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Duplicate Job: import_daily - 09/2016 size 156 with 0 attachments. Task: Job import_daily
2016-09-16 07:01:36 CEST Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Duplicate Job: import_daily - 09/2016 size 156 with 0 attachments. Task: Job import_daily
2016-09-18 07:01:37 CEST Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Duplicate Job: import_daily - 09/2016 size 156 with 0 attachments. Task: Job import_daily
2016-09-29 07:01:36 CEST Util.log_and_deliver_mail to=["ngiger@ywesee.com", "zdavatz@ywesee.com"] subject ch.ODDB.org Report - Duplicate Job: import_daily - 09/2016 size 156 with 0 attachments. Task: Job import_daily

I think that here Ruby does not handle the threads correctly. Will create a patch that should eliminate the problem and/or help debugging it.

  • replace the module global variable do_tracing by global variable $swissemedic_do_tracing in src/plugin/swissmedic.rb
  • set the $swissemedic_do_tracing twice with a sleep 0.1 in between
  • ensure that $swissemedic_do_tracing is set
  • add log entries after each of these actions

I am having problem running import_daily, because yesterday I had emerged a new version imagemagick. Error is 

  /usr/local/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_require.rb:73:in `require': This installation of RMagick was configured with ImageMagick 6.9.4 but ImageMagick 6.9.5-10 is in use. (RuntimeError)

Trying to reinstall the imagemagick gem fails with 

  sudo /usr/local/bin/gem193 install imagemagick
  ERROR:  Could not find a valid gem 'imagemagick' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=error: certificate verify failed (https://api.rubygems.org/specs.4.8.gz)

This fails with /usr/local/bin/gem193, /usr/local/bin/gem193p547 and /usr/local/bin/gem231

Calling manually openssl s_client -connect rubygems.org:443 I get 

<..>
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=l.ssl.fastly.net
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
---
No client certificate CA names sent
---
SSL handshake has read 5141 bytes and written 453 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: 5CAC84C84B677BFBD35C87180DFC5319FEF81B0C24820228FC2E708F7E4F161F
    Session-ID-ctx: 
    Master-Key: 2F2EFB02ADA00AF56AEEF90831043FECD4486CAF2655E5C01E4BCE3EF8CE07D589E9EB0D4B2CC3C4B1E04E360726CB6B
    Key-Arg   : None
    Start Time: 1475654307
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

I am trying whether upgrading openssl works. After sudo emerge openssl, calling openssl s_client -connect rubygems.org:443 -CAfile /etc/ssl/certs/ca-certificates.crt returns 

-----END CERTIFICATE-----
subject=/C=US/ST=California/L=San Francisco/O=Fastly, Inc./CN=l.ssl.fastly.net
issuer=/C=BE/O=GlobalSign nv-sa/CN=GlobalSign Organization Validation CA - SHA256 - G2
---
No client certificate CA names sent
---
SSL handshake has read 5141 bytes and written 453 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: C2483806ADA64AE72DD6C30FA4268E97D17E42C2BB685C766364ABE051B2922C
    Session-ID-ctx: 
    Master-Key: BF573DFA90CC5089A1B7AC2D99EF5EE58AC88458CB5A764BC4BDFDC7BCB05A4F2641A83C4FD7F513229F03005E9D10C0
    Key-Arg   : None
    Start Time: 1475654699
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)

Regenerating Ruby 2.3.1 to see whether we can install gems afterwards. Building fails now with 

Makefile:301: die Regel für Ziel „ossl_ssl.o“ scheiterte
make[2]: *** [ossl_ssl.o] Fehler 1
make[2]: Verzeichnis „/usr/local/src/ruby-2.3.1/ext/openssl“ wird verlassen
exts.mk:209: die Regel für Ziel „ext/openssl/all“ scheiterte
make[1]: *** [ext/openssl/all] Fehler 2
make[1]: Verzeichnis „/usr/local/src/ruby-2.3.1“ wird verlassen
uncommon.mk:203: die Regel für Ziel „build-ext“ scheiterte
make: *** [build-ext] Fehler 2

Remarked that I have an "OpenSSL 0.9.8zb 6 Aug 2014" installed under /usr/local/bin. Removing it via sudo rm -rf /usr/local/include/openssl /usr/local/bin/openssl /usr/local/ssl.

Added the lines 

  require 'openssl'
  OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

to /usr/local/lib/ruby/2.3.0/resolv.rb to get rid of the error.

Added the same lines to /usr/local/lib/ruby/1.9.1/resolv.rb to silence errors while calling sudo /usr/local/bin/gem193 install rmagick

But now I get some security warning when calling import_daily.

Pushed commit Avoid duplicate jobs error

Trying to port sbsm/davaz.com to use the rack webserver

Adapting sbsm/request.rb to use https://github.com/rack/rack. This is a lot easier if we change SBSM in a API incompatible way. Maybe we should clean up more stuff and create a new version 2.0 of SBSM which would rack and be no longer usable via (the outdated) mod_ruby.

Changing SBS::Request.new from initialize(drb_uri, html_version = "html4", cgiclass = CGI) to initialize(drb_uri, env, html_version = "html4")

By changing the module in index.rbx to 

module Rack
  class DavazCom
    def call(env)
      sbsm = SBSM::Request.new(DaVaz.config.server_uri, env)
      res = sbsm.process
      resp = Response.new
      resp.write res.to_s
      resp.finish
      resp
    end
  end
end

I am able to visit http://localhost:9191/ (or http://localhost:8751/) and getting an error response.

In the console where I called sudo -u apache /usr/local/bin/bundle exec doc/index_rack.rb I get the following output 

[2016-10-05 09:00:48] INFO  WEBrick 1.3.1
[2016-10-05 09:00:48] INFO  ruby 2.3.1 (2016-04-26) [x86_64-linux]
[2016-10-05 09:00:48] INFO  WEBrick::HTTPServer#start: pid=31672 port=9191
in process
StandardError
127.0.0.1 - - [05/Oct/2016:09:00:52 CEST] "GET / HTTP/1.0" 200 88
- -> /
in process
StandardError
::1 - - [05/Oct/2016:09:00:52 CEST] "GET /favicon.ico HTTP/1.0" 200 88
http://localhost:8751/ -> /favicon.ico

Pushed a commit Handling a request via rack creates StandardError

Leaving this task aside for the moment.

port sandoz.xmlconv to rack

As we are having problem with sandoz.xmlconv on fastpower, I must port it to use rack. Using sbsm via :path => in Gemfile. Calling /usr/local/bin/bundle install --path vendor; /usr/local/bin/bundle exec ruby doc/index.rbx and testing then test it via browser URL http://localhost:8181/.

Changing the implementation of sbsm/request.rb to handle Rack::Request instead of Apache.Request. Eg. Must replace @request.headers_out.add('Content-Disposition', "#@disposition; filename=#{basename}") by @request.set_header('Content-Disposition', "#@disposition; filename=#{basename}").

Now it begins to display some HTML content:

Links to the css and the images are wrong and do not contain the URL. E.g. we find href="http://resources/bbmb/bbmb.css" instead of href="http://localhost:8181/resources/bbmb/bbmb.css"

To test the soap I use curl -v http://localhost:8181/propharma -d@confuse.xml. The response received is wrong, as it is handled by index.rbx. Refactoring index.rbx to handle propharma, and keeping an eye that we have similar files for sunstore, etc.

Now the curl returns almost immediately saying 

curl -v http://localhost:8181/propharma -d@confuse.xml
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8181 (#0)
> POST /propharma HTTP/1.1
> Host: localhost:8181
> User-Agent: curl/7.50.3
> Accept: */*
> Content-Length: 625
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 625 out of 625 bytes

Landed in the pry debug and must do the correct handling. Adapted handling the post command to the rack environmen. But I get an error in the last line I must execute [xmlconv.execute_with_response(transaction) which throws me the following error 

  NoMethodError: undefined method `execute_with_response' for BBMB::Persistence::ODBA:BBMB::Util::Server
  from (druby://localhost:12004) /usr/local/lib/ruby/2.3.0/delegate.rb:87:in `method_missing'

Why?

If I call in the pry debugger transaction.execute, the call succeeded after I added 

  require 'conversion/pharmacieplus_bdd'
  require 'conversion/bdd_csv'
  require 'conversion/sunstore_bdd'

to the doc/index.rbx.

Debugging the response revelead that I get 

[7] pry(#<Rack::SandozXmlConv>)> transaction.response
NoMethodError: undefined method `deliveries' for nil:NilClass
from /home/niklaus/git/sandoz.xmlconv.bbmb.ch/lib/conversion/sunstore_bdd.rb:42:in `respond'

which is the same error as seen in the error_log on fastpower. I am asking myself where the confuse.xml is correct or whether it points to an inexisting order.

Zeno made the fastpower work again. Must look next week at http://dev.ywesee.com/Yasu/20160719-fix-import-error-sandoz-xmlconv-ch-and-remove-flavor-bbmb-ch

view · edit · sidebar · attach · print · history
Page last modified on October 05, 2016, at 07:52 PM