view · edit · sidebar · attach · print · history

< Niklaus.20170109-atc-ddd-update | Index | Niklaus.20161221-davaz-com-multithreaded >>

20161224-davaz-com-load-balancing

Summary

  • experimented with docker, nginx and load-balancing
  • Keep in Mind

Commits

Index

experimented with docker, nginx and load-balancing

As Zeno remarked, that performance (good interactive response time the most important) will be a major issue in 2017 for oddb.org, I kept thinking about whether using separat docker instances and using a load balancer would be a possible (at least partial) solution I decided to experiment on my own time and resources.

I used a two CPU virtual server from Hetzner (around 10 Euro/month), which I use as testbed for elexis. Therefore it can be reached via testbed-elexis.dyndns.org.

I decided to run two docker instances from davaz.com and using the nginx Debian package as load balancer.

The final setup consists of the following files:

Missing pieces:

  • YUS-Server
  • Sending Mails
  • Javascript don't work as expected. NodeJS setup??

Measurements:

wrk -t12 -c100 -d30s http://davaz.testbed-elexis.dyndns.org/resources/images/init/photo_davaz.jpg
Running 30s test @ http://davaz.testbed-elexis.dyndns.org/resources/images/init/photo_davaz.jpg
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    60.54ms   88.93ms   1.14s    90.87%
    Req/Sec   206.20     48.03   330.00     70.96%
  73782 requests in 30.10s, 384.06MB read
Requests/sec:   2451.43
Transfer/sec:     12.76MB

as compared to serve via Ruby's webrick davaz.com

niklaus@ng-tr /o/s/davaz.com> wrk -t12 -c100 -d30s http://davaz.testbed-elexis.dyndns.org/
Running 30s test @ http://davaz.testbed-elexis.dyndns.org/
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   318.00ms  288.19ms   1.99s    83.96%
    Req/Sec    30.02     16.59   111.00     62.32%
  10253 requests in 30.10s, 14.27MB read
  Socket errors: connect 0, read 0, write 0, timeout 3
Requests/sec:    340.65
Transfer/sec:    485.34KB

Using a slighly modified setup with only one davaz webrick servers lowers throuput from 340 to 220 request/seconds

Running 30s test @ http://davaz.testbed-elexis.dyndns.org/
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   428.67ms  212.33ms   1.69s    70.52%
    Req/Sec    19.78     10.57    70.00     87.86%
  6734 requests in 30.10s, 9.37MB read
  Socket errors: connect 0, read 0, write 0, timeout 1
Requests/sec:    223.73
Transfer/sec:    318.78KB

We get timeouts when running too many parallel requests. Also request/seconds handled drops drastically when using different URLs

niklaus@ng-tr /o/s/davaz.com> wrk -t12 -c100 -d30s --script=test/wrk_davaz.lua http://davaz.testbed-elexis.dyndns.org
Running 30s test @ http://davaz.testbed-elexis.dyndns.org
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   202.33ms  315.56ms   1.99s    93.92%
    Req/Sec     9.83      7.01    40.00     62.05%
  2160 requests in 30.04s, 79.53MB read
  Socket errors: connect 0, read 0, write 0, timeout 976
Requests/sec:     71.91
Transfer/sec:      2.65MB

On davaz.com where we have 8 CPU/16 Threads running we have less requests/second served

wrk -t12 -c100 -d30s --script=test/wrk_davaz.lua http://davaz.com
Running 30s test @ http://davaz.com
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   168.82ms  285.84ms   1.88s    88.84%
    Req/Sec    11.05      9.52    40.00     73.32%
  1119 requests in 30.04s, 50.05MB read
  Socket errors: connect 0, read 0, write 0, timeout 483
Requests/sec:     37.26
Transfer/sec:      1.67MB

Conclusion

  • The setup described works, well. nginx error_log contains no evil stuff
  • Using wrk -t12 -c100 brings the CPU-load on both core to >= 90%
  • Using wrk -t12 -c10000 provokes error with nginx, eg. not enough worker processes (limited to 768)
  • Using nginx to serve static files directly is very efficient.
  • Using two docker instances improves, but not doubles performance
  • It should be easy to setup a load balancing using several machines
  • It should be easy to setup specialized containers serving exactly one (or several) hostname/request types (e.g. for handling crawler)
  • To setup many (e.g. > 3) concurrent docker instances one should probably looks at tools like docker swarm
  • It would be easy to setup another app using a different MySQL server based on another MySQL version (e.g. 5.5 instead of 5.7).
  • Making nginx to respond nicely in presence of high (> 100) parallel request (eg. Denial of service) would need some care. But I think apache also offers tools for handling load balancing, see https://httpd.apache.org/docs/2.4/mod/mod_proxy_balancer.html.
  • One has to be very careful in interpreting results and look at what they measure really
view · edit · sidebar · attach · print · history
Page last modified on December 24, 2016, at 11:59 AM