Index>
20170111-oddb-org-ruby-240
Summary
- Port oddb.org to Ruby 2.4.0
- Create wrk/lua script for performance tests
- Keep in Mind
Commits
Index
Port oddb.org to Ruby 2.4.0
Comparing output of sudo -u apache bundle exec ruby2.4.0 jobs/import_bsv and sudo -u apache bundle exec ruby2.4.0 sudo -u apache bundle exec ruby2.4.0 jobs/import_swissmedic@@ with the output on thinpower.
jobs/import_swissmedic produced the following mails
- ch.ODDB.org Report - ATC-less - 01/2017 Total Sequences without ATC-Class: 93
- ch.ODDB.org Report - Refdata - 01/2017 Checked 24283 packages, Updated in trade (out_of_trade:false): 243 packages ...
- ch.ODDB.org Report - Co-Marketing - 01/2017 Found 610 Co-Marketing-Pairs of which 572 were found in the Database
- ch.ODDB.org Report - Patents - 01/2017 Checked 130 Registrations, Found 12 Patents, NotFound: 00373...
- ch.ODDB.org Report - Lppv - 01/2017 Updated Packages (lppv flag true): 0 details: Packages with SL-Entry: 0 Not updated were: 0 details:
- ch.ODDB.org Report - Error: Medwin-Companies - 01/2017 (Error: RuntimeError Message: could not connect to www.medwin.ch /refdata_wa_medwin/frmSearchPartner.aspx?lang=de: #<Net::HTTPMovedPermanently:)
- ch.ODDB.org Report - Export patents.xls - 01/2017 (with data/downloads/patents.xls)
jobs/import_bsv produced the following mails
- ch.ODDB.org Report - Generikaliste - 01/2017
- ch.ODDB.org Report - Med-Drugs - Swissmedic 12/2016 - SL 01/2017 swissdrug update Swissmedic 12/2016 - SL 01/2017
- ch.ODDB.org Report - oddb2.csv - 01/2017
- ch.ODDB.org Report - oddb.csv - 01/2017
- ch.ODDB.org Report - Lppv - 01/2017
- ch.ODDB.org Report - Refdata - 01/2017
- ch.ODDB.org Report - SL-Update (XML) - 01/2017 Am 10.01.2017 haben wir Ihren aktuellen SL-Export (XML) wieder überprüft. Dabei ist uns folgendes aufgefallen:...
- ch.ODDB.org Report - SL-Update (XML) - 01/2017 Created SL-Entries 105, Updated SL-Entries 9315 ...
Starting sudo -u apache bundle exec ruby2.4.0 jobs/import_daily. It finished without problems and sent the usual mails (recall, Fach- und Patienteninfo Updates)
Comparing generated artifacts of the mails mentioned here.
- /var/www/oddb.org/data/xls/med-drugs-20170110.xls with med-drugs-20170105.xls vom thin. Here I am missing entries for newly generated registrations like 65890 which was entered on January 10. Should I have to restart oddbd before or having called import_daily before? NO. They show different periods
swissdrug update Swissmedic 12/2016 - SL 01/2017 and swissdrug update Swissmedic 11/2016 - SL 01/2017
- /var/www/oddb.org/data/downloads/generics.xls. Cover different number of drugs. Checked Valsartan Actavis 40 mg, Filmtabletten which look the same.
- /var/www/oddb.org/data/downloads/oddb2.csv. Looks fairly similar, but quite a few differences probably due to different database content
- Same situation for oddb.csv
- Duplicate_Registrations_in_SL_10.01.2017.txt: Similar to thin
- Missing_Swissmedic_Codes_in_SL_10.01.2017.txt: Similar to thin
- Missing_Swissmedic_Codes_in_SL__out_of_trade__10.01.2017.txt: Similar to thin
- Unknown_Products_in_SL_10.01.2017.txt: Similar to thin
- Package_Data_was_completed_from_SL.txt: Similar to thin
- Package_Data_was_completed_from_SL.txt: Same (except time stamp) to thin
- /var/www/oddb.org/data/downloads/patents.xls. Directlink is different. Is the setup on oddb-ci2 not correct??
Updated Attach:gen_ruby_240.txt to create logical links for ruby-240 and bundle-240.
Installed ruby-240 on thinpower.
Create wrk/lua script for performance tests
Working on creating a wrk script that should guarantee
- login as desitin
- visit 20 FI
- visit FI diff
- visit Teva, Novartis, Sandoz
- visit price comparision for Iscador, Aspirin
Also will fix the install script for Ruby 2.4.0 to install bundler and create a bundle-2.4.0 command.
Now my wrk script starts to check various error conditions. Snippet is
function response(status, headers, body)
local msg = "body %s\nunexpected status %d for id %d path %s"
local okay = "thread id %d path %s"
local not_found = 'Die von Ihnen gewünschte Information ist leider nicht mehr vorhanden.'
local limit = 'Abfragebeschränkung'
if (string.find(body, limit)) then
print(msg:format(body, status, id, path))
print(limit)
os.exit(status)
end
if (string.find(body, not_found)) then
print(msg:format(body, status, id, path))
print(not_found)
os.exit(status)
end
if (status == 200) then
print(okay:format(id, path))
else
print(msg:format(body, status, id, path))
os.exit(status)
end
if (id == 1) then os.execute("sleep 0.5") end
responses = responses + 1
end
Now I must work that each wrk thread does first a login. But tried in vain on howto setup login for desitin. Should we use something like http://czerasz.com/2015/07/19/wrk-http-benchmarking-tool-example/?
No. We just disable the login test on the test server.
Now the log file is quite instructive as it show the visited path and the time it took for loading the result. See Attach:wrk_performance_log.txt. Also it shows now a much more realistic nr of request per seconds Requests/sec: 2.79
Wasted half an hour to try to get calculated the elapsed time with a resolution better than 1 second. But wrk/lua does not offer out of the box working solution.
Pushed commit Added lua script for performance testing
Included the following remarks in lua script:
- show visited path and time taken for it (varies greatly!)
- use a typical mix of visited
- thread nr 2 sleeps 2 seconds after each request (just to be sure, that this thread must have less requests
- response with status != 200 abort the script
- response which contains 'Die von Ihnen gewünschte Information ist leider nicht mehr vorhanden.' or 'Abfragebeschränkung' abort the script
- must use patched test site which
- returns '' in the method to_html of src/view/google_ad_sense.rb
- returns false in the method limited? of src/state/global.rb
- TODO: cleanup correctly, because running it several time the server crashes with an error message like
With 2007 sessions we have more than 2000 sessions. Exiting