wiki:PerformanceTesting

Introduction

There have been significant changes in the future 2.3 branch since the relatively stable and conservative 2.1.x and 2.2.x releases. Therefore there's a definite need for wide testing in various areas, including performance. This page tries to outline what the most relevant tests are so that they can be ran on a regular basis during development to spot any regressions and bottlenecks. These tests are primarily aimed at the future OpenVPN 2.3 branch (current "master"). However, same setup can used to test older versions, too. Focus is on testing the most common and/or potentially useful configurations, as testing all combinations is practically impossible.

Goals

The tests have several goals:

  • Get an idea of OpenVPN performance with any given parameters
  • Identify performance bottlenecks and fix the underlying issues insofar as possible
  • Identify non-linear resource comsumption growth, e.g. find out if 100 clients consumes (significantly) more resources than that of 10 clients (multiplied by ten)
  • Detect and fix abnormal behavior (e.g. partially configured clients) during peak load periods

When these tests are repeated over time, they also allow spotting regressions.

Prequisites

Identification of performance bottlenecks requires plotting resource consumption, e.g. memory usage and processor load. This, in turn may require selective adding of new debugging code, so that excessively verbose, generic logging does not degrade performance and thus produce biased results. Maximum performance tests don't have this requirement, as they only need minimal logging. In addition, spotting and debugging configuration errors requires collecting information about OpenVPN's state on the client side.

There are further requirements, depending on the test environments being used; see below for details.

Test cases

Here we should have a high-level list of test cases. Technical details (e.g. OpenVPN client configuration files) can also be attached. Mattock can then convert these descriptions into Fabric code which makes the magic happen.

Throughput tests

There are a number of factors affecting throughput:

  • number of clients
  • bandwidth per client
  • encryption algorithms/ciphers
  • compression
  • packet size
  • transport protocol (TCP/UDP)
  • network errors
  • others factors, which?

There are a few specific questions that need an answer (feel free to add more):

  1. Maximum throughput using various parameters
  2. How does (excessive) packet loss affect performance?
  3. Does number of clients affect performance, if data transfer rate remains the same? (or "non-linear resource consumption growth")

Below is a list of test cases:

Test nameClientsAnswer to
MP-151,3
MP-2201,3
MP-3501,3
MP-41001,3

Protocol needs to be UDP, as TCP (in iperf) prevents controlling throughput. Each test will be repeated for each of the following combinations:

UDPIPv4IPv6OpenSSLPolarSSLOpenVPN 2.2.1OpenVPN (master)
XX X X
XX X X
X XX X
X X X X
XX X

Question: Should a subset of these tests repeated for TCP?

Connection tests

Connection tests are aimed at identifying bottlenecks during spikes of (re)connection initiations, e.g. when 100 clients try to connect to the server simultaneously.

Insofar as possible, the effects of various configuration parameters are tested; especially those which generate extra traffic between server to client.

A few specific questions:

  1. Max number of (re)connections per second before server chokes (or, expensiveness of connection establishment)
  2. What happens _when_ server is overloaded?
    • Do the clients connect properly after the server starts responding again? Or do they lose connectivity until they are restarted?
    • Do all the clients end up in expected state if the server is overloaded during connection initialization?
    • Are all client configuration directives (e.g. routes) pushed properly?
  3. Reconnections
    • Are these any different from initial handshakes?

Proposed test cases are shown below:

Test nameVM x OpenVPN instancesAnswer to
C-150x21,2?
C-250x61,2?
C-350x101,2?

The least powerful instances in EC2 (e.g. m1.small) can be slowed down to a crawl fairly easily with 50-100 (real) clients. For more than 100 simultaneous connections several parallel OpenVPN processes are used on the same client instance; although the clients will fail to initialize properly, they should still stress the server in relatively realistic fashion.

Number of simultaneous concurrent connections is increased until server chokes. Each test will be repeated for the following configurations:

IPv4IPv6OpenSSLPolarSSLOpenVPN 2.2.1OpenVPN (master)
X X X
X X X

Question: does IP protocol version or SSL backend affect initial connection establishment significantly?

Test results

Test setup details for m1.small servers

All server instances on EC2 were the same:

  • AMI: OpenVPN Access Server 1.8.3 (ami-31d21358)
  • Operating system: Ubuntu 11.04 "Natty Narwhal" (i386)
  • Kernel: 2.6.38-8-virtual #42-Ubuntu SMP (aki-407d9529)

Client instances on EC2 used a slightly different setup:

  • AMI: Ubuntu 10.04 (ami-81b275e8)
  • Operating system: Ubuntu 10.04 "Lucid Lynx" (i386)
  • Kernel: 2.6.32-317-ec2 #36-Ubuntu SMP (aki-407d9529)

All EC2 instances were created to the same availability zone (us-east-1d). The setup of 100 client instances took ~4 minutes with 34 threads running Fabric (~3 runs per thread).

Test setup details for m1.large servers

All server instances on EC2 were the same:

  • AMI: ami-fd589594
  • Operating system: Ubuntu 11.04 "Natty Narwhal" (amd64)
  • Kernel: 2.6.38-11-virtual #50-Ubuntu SMP (aki-427d952b)

Client instances on EC2 used a slightly different setup:

  • AMI: Ubuntu 10.04 (ami-81b275e8)
  • Operating system: Ubuntu 10.04 "Lucid Lynx" (i386)
  • Kernel: 2.6.32-317-ec2 #36-Ubuntu SMP (aki-407d9529)

All EC2 instances were created to the same availability zone (us-east-1d). The setup of 100 client instances took ~4 minutes with 34 threads running Fabric (~3 runs per thread).

MP-2 results

OpenVPN 2.2.1/m1.large/UDP/IPv4/OpenSSL/20 clients (MP-2)

Test names:

  • 24-mp2.m1.large-openvpn221-1daemon-4iperfs
  • 25-mp2.m1.large-openvpn221-1daemon-4iperfs

Connection establishment

Connection establishment took 1-2 seconds, with peak CPU load at 3-5%.

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
24-mp2.m1.large-openvpn221-1daemon-4iperfs120saverage182181.717.980.998889
25-mp2.m1.large-openvpn221-1daemon-4iperfs120saverage202252.3618.510.9255

Server statistics

TestSubtestReal lengthCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
24-mp2.m1.large-openvpn221-1daemon-4iperfs120s1308.7082315.87429.80970.0595538
25-mp2.m1.large-openvpn221-1daemon-4iperfs120s1328.6559515.796929.80140.0505076

MP-3 results

OpenVPN 2.2.1/m1.large/UDP/IPv4/OpenSSL/50 clients (MP-3)

Test names:

  • 26-mp3.m1.large-openvpn221-1daemon-4iperfs
  • 27-mp3.m1.large-openvpn221-1daemon-4iperfs

Connection establishment

Connection establishment took 2-3 seconds, with peak CPU load at 11-12%.

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
26-mp3.m1.large-openvpn221-1daemon-4iperfs120saverage462281.1818.120.393913
27-mp3.m1.large-openvpn221-1daemon-4iperfs120saverage452334.1417.960.399111

Server statistics

TestSubtestReal lengthCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
26-mp3.m1.large-openvpn221-1daemon-4iperfs120s1308.7250416.199430.40420.0720923
27-mp3.m1.large-openvpn221-1daemon-4iperfs120s1318.954615.89330.42520.0520763

MP-4 results

Test environments

General considerations

In environments that are not isolated (e.g. Amazon EC2) the same tests need to be run twice or more to see if external factors affect the results noticeably.

"It is important to use the iperf results from the *receiving* end, which is the side where you start the server (iperf -s) . The numbers from the sending party are not particularly reliable, they tend to fluctuate much more, esp over UDP. This makes some sense, as the sending party spits out packets at a particular rate and it does not care whether they arrive or not. This will skew the results slightly." (source)

Amazon EC2

Amazon EC2 is a service that can be used to generate lots of VMs, e.g. for performance testing purposes. There are several different types of instance types (=VMs) available. The pricing pricing is divided into hourly rent and bandwidth costs. The latter only apply to traffic going out of the EC2. So, in order to save bandwidth costs, the whole test environment can be built to EC2 within the same availability zone.

Amazon EC2 is used for maximum simultaneous connection testing, where a large number of clients is required. It is also - for now at least - used for maximum throughput tests.

Test environment setup is handled by the Python deployment tool (see below) that utilizes Boto libraries and Fabric under the hood.

Local (LAN)

LAN environment is/can be used for throughput tests. Server and clients are connected to a high-quality, high-bandwidth switch to avoid forming external bottlenecks.

Test environment setup is handled by the Python deployment tool that utilizes Fabric under the hood. Initial installation of client OSes is automated using FAE (Debian) or Kickstart (RHEL/Fedora derivatives).

Configuring test server(s)

All the logic is embedded into the deployment tool.

Configuring test clients

All the logic is embedded into the deployment tool.

Tools

Deploying clients

Mattock (on IRC) has written a small application that eases deployment of large number of OpenVPN client computers that can adequately stress a server. At the moment (Sep 2011) the app is used for Amazon EC2 deployments, but it's pretty general purpose and easily extendable. It's also not fully functional yet, as one of it components (Fabric) does not play well in this kind of scenarios with updating and/or small modifications. For details, se the included README file.

To get this tool (perftest), do the following:

$ wget http://swupdate.openvpn.net/downloads/perftest.tar.gz

The README file contains everything you need to know. At least initially patches should be sent to samuli at openvpn and net, or to mattock who hangs around at #openvpn-devel on the Freenode IRC network (irc.freenode.net).

Generating network traffic

Iperf can be used to generate network traffic. It has previously been used to tune OpenVPN performance: look here. There are a few caveats when using iperf as described by Jason Haar (original message on openvpn-users mailing list).

FWIW iperf doesn't send random data - it's sends the ASCII sequence over
and over again. i.e. it is "compressible" depending on what lies
between. We have Riverbeds here and I was amazed how our 10Mbs WAN links
suddenly jumped to 1Gbs according to iperf! Ended up the Riverbeds were
able to do a lot of smoke-n-mirrors with iperf traffic

You can run iperf as "iperf -c ser.ver.name -F /tmp/big.zip.file" so
that it is actually sending (effectively) random data - in order to
remove some of the potential tricks that could lie between your client
and server.

Also, "iperf -c ser.ver.name -P 4" is a good way of measuring BANDWIDTH
instead of THROUGHPUT - which a standard single-channel TCP stream measures

Collecting data

Dstat can be used to gather detailed resource utilization statistics on the server side.

Notes

Head/tail effect

Initial tests with m1.small server instances, all test phases (connection -> 10s -> 30s -> 60s) were launched from the same (serially executed) shell script, so any delays propagated to later phases. In later tests that used m1.large server instances each subtest was precisely timed using at, so this propagation of delays was eliminated. Regardless of this the delays remained, meaning that a 10 second test could easily become a 20+ second test. The culprit was not the serial nature of timing, but the incapability of the server processes - especially iperf - to establish new connections quickly started to show with as few as 20 clients. This was clearly visible when looking at server-side iperf output: clients connected with delay of several seconds. This had two effects:

  • Prolonging the subtests (e.g. 10 second test could become 20+ second test)
  • Artificially increased the average client bandwidth due to the head and tail of clients that had less competition (bandwidth-vise)

OpenVPN connection initialization seems to be much faster.

This head-tail effect can skew the results badly. These test results are a good example: two identical tests can produce quite different results. There are a few ways to counter this effect:

  • Lengthen the iperf test periods, so that any delays have a less pronounced effect
  • Randomize the client connection to iperf
  • Launch several iperf instances and split client connections between these

These changes combined have produced more reliable results test after test.

Last modified 20 months ago Last modified on 08/26/15 06:56:32