Index>

20161224-davaz-com-load-balancing

Summary

• experimented with docker, nginx and load-balancing
• Keep in Mind

Commits

Index

• experimented with docker, nginx and load-balancing

experimented with docker, nginx and load-balancing

As Zeno remarked, that performance (good interactive response time the most important) will be a major issue in 2017 for oddb.org, I kept thinking about whether using separat docker instances and using a load balancer would be a possible (at least partial) solution I decided to experiment on my own time and resources.

I used a two CPU virtual server from Hetzner (around 10 Euro/month), which I use as testbed for elexis. Therefore it can be reached via testbed-elexis.dyndns.org.

I decided to run two docker instances from davaz.com and using the nginx Debian package as load balancer.

The final setup consists of the following files:

• /etc/nginx/sites-enabled/davaz.com. See Attach:davaz.com.nginx.txt
• a checkout of github.com/ngiger/davaz.com with the added files as
  • docker/Dockerfile: See Attach:davaz.com.Dockerfile.txt
  • docker/docker-compose.yml: See Attach:docker-compose.yml.txt
  • docker/config.yml (no YUS-Server, no valid SMTP). See Attach:davaz.com.docker.config.txt
  • docker/db_connection_data.yml to describe the connection to the mysql docker instance: See Attach:davaz.com.docker.db_connection_data.txt
  • docker/dump with the bak.mysql.2016-12-21_22.00.36_davaz2*.sql files to populate the MySQL docker db
  • a slightly patche src/util/app.rb to avoid connection to the (absent) YUS-Server
• nginx configured to
  • serve static files directly for URL starting with /resources
  • use two instances of davaz.com container

Missing pieces:

• YUS-Server
• Sending Mails
• Javascript don't work as expected. NodeJS setup??

Measurements:

wrk -t12 -c100 -d30s http://davaz.testbed-elexis.dyndns.org/resources/images/init/photo_davaz.jpg
Running 30s test @ http://davaz.testbed-elexis.dyndns.org/resources/images/init/photo_davaz.jpg
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    60.54ms   88.93ms   1.14s    90.87%
    Req/Sec   206.20     48.03   330.00     70.96%
  73782 requests in 30.10s, 384.06MB read
Requests/sec:   2451.43
Transfer/sec:     12.76MB

as compared to serve via Ruby's webrick davaz.com

niklaus@ng-tr /o/s/davaz.com> wrk -t12 -c100 -d30s http://davaz.testbed-elexis.dyndns.org/
Running 30s test @ http://davaz.testbed-elexis.dyndns.org/
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   318.00ms  288.19ms   1.99s    83.96%
    Req/Sec    30.02     16.59   111.00     62.32%
  10253 requests in 30.10s, 14.27MB read
  Socket errors: connect 0, read 0, write 0, timeout 3
Requests/sec:    340.65
Transfer/sec:    485.34KB

Using a slighly modified setup with only one davaz webrick servers lowers throuput from 340 to 220 request/seconds

Running 30s test @ http://davaz.testbed-elexis.dyndns.org/
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   428.67ms  212.33ms   1.69s    70.52%
    Req/Sec    19.78     10.57    70.00     87.86%
  6734 requests in 30.10s, 9.37MB read
  Socket errors: connect 0, read 0, write 0, timeout 1
Requests/sec:    223.73
Transfer/sec:    318.78KB

We get timeouts when running too many parallel requests. Also request/seconds handled drops drastically when using different URLs

niklaus@ng-tr /o/s/davaz.com> wrk -t12 -c100 -d30s --script=test/wrk_davaz.lua http://davaz.testbed-elexis.dyndns.org
Running 30s test @ http://davaz.testbed-elexis.dyndns.org
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   202.33ms  315.56ms   1.99s    93.92%
    Req/Sec     9.83      7.01    40.00     62.05%
  2160 requests in 30.04s, 79.53MB read
  Socket errors: connect 0, read 0, write 0, timeout 976
Requests/sec:     71.91
Transfer/sec:      2.65MB

On davaz.com where we have 8 CPU/16 Threads running we have less requests/second served

wrk -t12 -c100 -d30s --script=test/wrk_davaz.lua http://davaz.com
Running 30s test @ http://davaz.com
  12 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   168.82ms  285.84ms   1.88s    88.84%
    Req/Sec    11.05      9.52    40.00     73.32%
  1119 requests in 30.04s, 50.05MB read
  Socket errors: connect 0, read 0, write 0, timeout 483
Requests/sec:     37.26
Transfer/sec:      1.67MB

Conclusion

• The setup described works, well. nginx error_log contains no evil stuff
• Using wrk -t12 -c100 brings the CPU-load on both core to >= 90%
• Using wrk -t12 -c10000 provokes error with nginx, eg. not enough worker processes (limited to 768)
• Using nginx to serve static files directly is very efficient.
• Using two docker instances improves, but not doubles performance
• It should be easy to setup a load balancing using several machines
• It should be easy to setup specialized containers serving exactly one (or several) hostname/request types (e.g. for handling crawler)
• To setup many (e.g. > 3) concurrent docker instances one should probably looks at tools like docker swarm
• It would be easy to setup another app using a different MySQL server based on another MySQL version (e.g. 5.5 instead of 5.7).
• Making nginx to respond nicely in presence of high (> 100) parallel request (eg. Denial of service) would need some care. But I think apache also offers tools for handling load balancing, see https://httpd.apache.org/docs/2.4/mod/mod_proxy_balancer.html.
• One has to be very careful in interpreting results and look at what they measure really