view · edit · sidebar · attach · print · history

< Niklaus.20170425-virbac-importer | Index | Niklaus.20170419-virbac >>

20170424-drugshortage-unable-to-parse

Summary

  • Fix error parsing drugshortage
  • Rate limit bitfinex
  • Port virbac to rack
  • Keep in Mind

Commits

Index

Fix error parsing drugshortage

We received again error messages with

Error: RuntimeError
Message: unable to parse https://www.drugshortage.ch/UebersichtaktuelleLieferengpaesse2.aspx
Backtrace:
/var/www/oddb.org/src/plugin/shortage.rb:145:in `update_drugshortage'
/var/www/oddb.org/src/plugin/shortage.rb:49:in `update'
/var/www/oddb.org/src/util/updater.rb:369:in `block in update_drugshortage'
/var/www/oddb.org/src/util/updater.rb:543:in `wrap_update'
/var/www/oddb.org/src/util/updater.rb:367:in `update_drugshortage'
/var/www/oddb.org/src/util/updater.rb:207:in `run'
jobs/import_daily:13:in `block in <module:Util>'
/var/www/oddb.org/src/util/job.rb:40:in `run'
jobs/import_daily:12:in `<module:Util>'
jobs/import_daily:11:in `<module:ODDB>'
jobs/import_daily:10:in `<main>'

Must find a way to reproduce this problem on oddb-ci2. Zeno suspects that it happens when we

  • use yesterday's file for Drugshortage (e.g. mv data/html/drugshortage-2017.04.13.html -> data/html/drugshortage-latest.html)
  • import_daily
  • use yesterday's file for Drugshortage
  • import_daily
  • use yesterday's file for Drugshortage
  • import_daily

Also some changes were not handled correctly. Eg.

  • the last update date for ANESDERM Creme 5 % 5 Tb 5 g
  • the new entry for ANESDERM Creme 5 % Tb 30 g is missing. In drugshortage-2017.04.24.html we find
<td nowrap="nowrap"><font face="Arial" color="Black" size="3"><a href="detail_lieferengpass.aspx?ID=3173">
ANESDERM Creme 5 % Tb 30 g</a>
</font></td>
<td><font face="Arial" color="Black" size="3">7680566820033</font></td>
<td><font face="Arial" color="Black" size="3">4092973</font></td>
<td nowrap="nowrap"><font face="Arial" color="Black" size="3">Pierre Fabre (Suisse) S.A.</font></td>
<td><font face="Arial" color="Black" size="3">21.04.2017</font></td><td align="center"><font face="Arial" color="Black" size="3">3</font></td>
<td><font face="Arial" color="Black" size="3">aktuell keine Lieferungen</font></td>
<td><font face="Arial" color="Black" size="3">28.04.2017</font></td>
<td><font face="Arial" color="Black" size="3"><a href="alternativen.aspx?GTIN=7680566820033">Alternativen?</a></font></td>

Found the reason for not updating some field. Did not check the contents of the downloaded html file. Created a new html test file for drug_shortage with some changed fields. Now I am getting in the unit test also the parse error. Which if fine, as first a test must fail!

Fixed the problem with commit Fix parsing drugshortage when only some fields changed

Here an example how only affected fields are shown in the email

DrugShortag changes:
7680656020039;N02CC06;Eletriptan-Mepha 40 mg, Lactab shortage_last_update: 2017-04-12 => 2017-04-24
              shortage_delivery_date: KW 19 => KW 22
7680656020022;N02CC06;Eletriptan-Mepha 40 mg, Lactab shortage_state:  => aktuell keine Lieferungen
              shortage_last_update:  => 2017-04-24
              shortage_delivery_date:  => KW 21
              shortage_link:  => https://www.drugshortage.ch/detail_lieferengpass.aspx?ID=3005

Rate limit bitfinex

We must limit the access via the bitfinex API to about 90 / minutes.

But first we fixe the error, that the unit tests did not complete under travis-CI with commit Create dummy Config when running RSpec tests

For getting the history date we must use the REST API, as there is not websocket API for it. It is rate limited to 15 request per minute. It looks that after hitting the limit you must wait at least 60 seconds till the next request get through. When sleeping 2.1 seconds after each request, I was able to read 237 times without any problem. But this is slow. Therefore trying an algorithm which blocks only when we hit the rate limit.

Pushed commits

Zeno remarked another timeout error, which I as able to see once on my machine, too. Eg.

Summary for deposit XMR (31 entries}
 Margin Funding Payment on wallet Deposit is 0.18818553999999998
                       Deposit (MONERO) # is 44.67225245
Fetching returned [:error, "ERR_RATE_LIMIT"]. Aborting

This happens in another code path. (And only the passing the -c flag to rapflag). Fixed the problems. Errors are show like

  Margin Funding Payment correction for  is 0.0050965099999999985

Summary for deposit XMR (31 entries}
 Margin Funding Payment on wallet Deposit is 0.18818553999999998
                       Deposit (MONERO) # is 44.67225245
2017-04-24 17:43:41 +0200: Fetching https://api.bitfinex.com/v2/candles/trade:1D:tXMRUSD/hist?start:1490396400000 returned [:error, "ERR_RATE_LIMIT"].
   Retrying in 30 seconds
2017-04-24 17:44:11 +0200: Fetching https://api.bitfinex.com/v2/candles/trade:1D:tXMRUSD/hist?start:1490396400000 returned [:error, "ERR_RATE_LIMIT"].
   Retrying in 30 seconds
2017-04-24 17:44:41 +0200: Fetching https://api.bitfinex.com/v2/candles/trade:1D:tXMRUSD/hist?start:1490396400000 returned [:error, "ERR_RATE_LIMIT"].
   Retrying in 30 seconds

Summary for deposit ZEC (26 entries}

It often takes two minutes till we can continue. Pushed commit Fix ERR_TIMOUT while fetching rates

Port virbac to rack

Continue work make login work correctly under rack. After some changes I get logged in as new customer instead of a known user. Why? When pressing logout after the (wrong) login I get an error
ArgumentError: wrong number of arguments (given 2, expected 3)
        (druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/yus-1.0.4/lib/yus/server.rb:24:in `login'
        (druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1624:in `perform_without_block'
        (druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1584:in `perform'
        (druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1668:in `block (2 levels) in main_loop'
        (druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `loop'
        (druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `block in main_loop'
        /home/niklaus/git/bbmb/lib/bbmb/html/util/session.rb:21:in `login'
        /home/niklaus/git/bbmb/lib/bbmb/html/state/login.rb:18:in `login'
        /home/niklaus/git/sbsm/lib/sbsm/state.rb:197:in `_trigger'
        /home/niklaus/git/sbsm/lib/sbsm/state.rb:186:in `trigger'
        /home/niklaus/git/sbsm/lib/sbsm/session.rb:217:in `block in process_rack'
        /home/niklaus/git/sbsm/lib/sbsm/session.rb:208:in `synchronize'
        /home/niklaus/git/sbsm/lib/sbsm/session.rb:208:in `process_rack'
        /home/niklaus/git/sbsm/lib/sbsm/app.rb:119:in `call'

The domain parameter is not passed. Why?

view · edit · sidebar · attach · print · history
Page last modified on April 24, 2017, at 05:49 PM