We received again error messages with
Error: RuntimeError Message: unable to parse https://www.drugshortage.ch/UebersichtaktuelleLieferengpaesse2.aspx Backtrace: /var/www/oddb.org/src/plugin/shortage.rb:145:in `update_drugshortage' /var/www/oddb.org/src/plugin/shortage.rb:49:in `update' /var/www/oddb.org/src/util/updater.rb:369:in `block in update_drugshortage' /var/www/oddb.org/src/util/updater.rb:543:in `wrap_update' /var/www/oddb.org/src/util/updater.rb:367:in `update_drugshortage' /var/www/oddb.org/src/util/updater.rb:207:in `run' jobs/import_daily:13:in `block in <module:Util>' /var/www/oddb.org/src/util/job.rb:40:in `run' jobs/import_daily:12:in `<module:Util>' jobs/import_daily:11:in `<module:ODDB>' jobs/import_daily:10:in `<main>'
Must find a way to reproduce this problem on oddb-ci2. Zeno suspects that it happens when we
Also some changes were not handled correctly. Eg.
<td nowrap="nowrap"><font face="Arial" color="Black" size="3"><a href="detail_lieferengpass.aspx?ID=3173"> ANESDERM Creme 5 % Tb 30 g</a> </font></td> <td><font face="Arial" color="Black" size="3">7680566820033</font></td> <td><font face="Arial" color="Black" size="3">4092973</font></td> <td nowrap="nowrap"><font face="Arial" color="Black" size="3">Pierre Fabre (Suisse) S.A.</font></td> <td><font face="Arial" color="Black" size="3">21.04.2017</font></td><td align="center"><font face="Arial" color="Black" size="3">3</font></td> <td><font face="Arial" color="Black" size="3">aktuell keine Lieferungen</font></td> <td><font face="Arial" color="Black" size="3">28.04.2017</font></td> <td><font face="Arial" color="Black" size="3"><a href="alternativen.aspx?GTIN=7680566820033">Alternativen?</a></font></td>
Found the reason for not updating some field. Did not check the contents of the downloaded html file. Created a new html test file for drug_shortage with some changed fields. Now I am getting in the unit test also the parse error. Which if fine, as first a test must fail!
Fixed the problem with commit Fix parsing drugshortage when only some fields changed
Here an example how only affected fields are shown in the email
DrugShortag changes:
7680656020039;N02CC06;Eletriptan-Mepha 40 mg, Lactab shortage_last_update: 2017-04-12 => 2017-04-24
shortage_delivery_date: KW 19 => KW 22
7680656020022;N02CC06;Eletriptan-Mepha 40 mg, Lactab shortage_state: => aktuell keine Lieferungen
shortage_last_update: => 2017-04-24
shortage_delivery_date: => KW 21
shortage_link: => https://www.drugshortage.ch/detail_lieferengpass.aspx?ID=3005
We must limit the access via the bitfinex API to about 90 / minutes.
But first we fixe the error, that the unit tests did not complete under travis-CI with commit Create dummy Config when running RSpec tests
For getting the history date we must use the REST API, as there is not websocket API for it. It is rate limited to 15 request per minute. It looks that after hitting the limit you must wait at least 60 seconds till the next request get through. When sleeping 2.1 seconds after each request, I was able to read 237 times without any problem. But this is slow. Therefore trying an algorithm which blocks only when we hit the rate limit.
Pushed commits
Zeno remarked another timeout error, which I as able to see once on my machine, too. Eg.
Summary for deposit XMR (31 entries}
Margin Funding Payment on wallet Deposit is 0.18818553999999998
Deposit (MONERO) # is 44.67225245
Fetching returned [:error, "ERR_RATE_LIMIT"]. Aborting
This happens in another code path. (And only the passing the -c flag to rapflag). Fixed the problems. Errors are show like
Margin Funding Payment correction for is 0.0050965099999999985
Summary for deposit XMR (31 entries}
Margin Funding Payment on wallet Deposit is 0.18818553999999998
Deposit (MONERO) # is 44.67225245
2017-04-24 17:43:41 +0200: Fetching https://api.bitfinex.com/v2/candles/trade:1D:tXMRUSD/hist?start:1490396400000 returned [:error, "ERR_RATE_LIMIT"].
Retrying in 30 seconds
2017-04-24 17:44:11 +0200: Fetching https://api.bitfinex.com/v2/candles/trade:1D:tXMRUSD/hist?start:1490396400000 returned [:error, "ERR_RATE_LIMIT"].
Retrying in 30 seconds
2017-04-24 17:44:41 +0200: Fetching https://api.bitfinex.com/v2/candles/trade:1D:tXMRUSD/hist?start:1490396400000 returned [:error, "ERR_RATE_LIMIT"].
Retrying in 30 seconds
Summary for deposit ZEC (26 entries}
It often takes two minutes till we can continue. Pushed commit Fix ERR_TIMOUT while fetching rates
ArgumentError: wrong number of arguments (given 2, expected 3)
(druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/gems/2.4.0/gems/yus-1.0.4/lib/yus/server.rb:24:in `login'
(druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1624:in `perform_without_block'
(druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1584:in `perform'
(druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1668:in `block (2 levels) in main_loop'
(druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `loop'
(druby://virbac.bbmb.ngiger.ch:12003) /usr/local/ruby-2.4.0/lib/ruby/2.4.0/drb/drb.rb:1664:in `block in main_loop'
/home/niklaus/git/bbmb/lib/bbmb/html/util/session.rb:21:in `login'
/home/niklaus/git/bbmb/lib/bbmb/html/state/login.rb:18:in `login'
/home/niklaus/git/sbsm/lib/sbsm/state.rb:197:in `_trigger'
/home/niklaus/git/sbsm/lib/sbsm/state.rb:186:in `trigger'
/home/niklaus/git/sbsm/lib/sbsm/session.rb:217:in `block in process_rack'
/home/niklaus/git/sbsm/lib/sbsm/session.rb:208:in `synchronize'
/home/niklaus/git/sbsm/lib/sbsm/session.rb:208:in `process_rack'
/home/niklaus/git/sbsm/lib/sbsm/app.rb:119:in `call'
The domain parameter is not passed. Why?