wiki:PerformanceTesting

Version 33 (modified by Samuli Seppänen, 12 years ago) (diff)

--

Introduction

There have been significant changes in the future 2.3 branch since the relatively stable and conservative 2.1.x and 2.2.x releases. Therefore there's a definite need for wide testing in various areas, including performance. This page tries to outline what the most relevant tests are so that they can be ran on a regular basis during development to spot any regressions and bottlenecks. These tests are primarily aimed at the future OpenVPN 2.3 branch (current "master"). However, same setup can used to test older versions, too. Focus is on testing the most common and/or potentially useful configurations, as testing all combinations is practically impossible.

Goals

The tests have several goals:

  • Get an idea of OpenVPN performance with any given parameters
  • Identify performance bottlenecks and fix the underlying issues insofar as possible
  • Identify non-linear resource comsumption growth, e.g. find out if 100 clients consumes (significantly) more resources than that of 10 clients (multiplied by ten)
  • Detect and fix abnormal behavior (e.g. partially configured clients) during peak load periods

When these tests are repeated over time, they also allow spotting regressions.

Prequisites

Identification of performance bottlenecks requires plotting resource consumption, e.g. memory usage and processor load. This, in turn may require selective adding of new debugging code, so that excessively verbose, generic logging does not degrade performance and thus produce biased results. Maximum performance tests don't have this requirement, as they only need minimal logging. In addition, spotting and debugging configuration errors requires collecting information about OpenVPN's state on the client side.

There are further requirements, depending on the test environments being used; see below for details.

Test cases

Here we should have a high-level list of test cases. Technical details (e.g. OpenVPN client configuration files) can also be attached. Mattock can then convert these descriptions into Fabric code which makes the magic happen.

Throughput tests

There are a number of factors affecting throughput:

  • number of clients
  • bandwidth per client
  • encryption algorithms/ciphers
  • compression
  • packet size
  • transport protocol (TCP/UDP)
  • network errors
  • others factors, which?

There are a few specific questions that need an answer (feel free to add more):

  1. Maximum throughput using various parameters
  2. How does (excessive) packet loss affect performance?
  3. Does number of clients affect performance, if data transfer rate remains the same? (or "non-linear resource consumption growth")

Below is a list of test cases:

Test nameClientsAnswer to
MP-151,3
MP-2201,3
MP-3501,3
MP-41001,3

Protocol needs to be UDP, as TCP (in iperf) prevents controlling throughput. Each test will be repeated for each of the following combinations:

UDPIPv4IPv6OpenSSLPolarSSLOpenVPN 2.2.1OpenVPN (master)
XX X X
XX X X
X XX X
X X X X
XX X

Question: Should a subset of these tests repeated for TCP?

Connection tests

Connection tests are aimed at identifying bottlenecks during spikes of (re)connection initiations, e.g. when 100 clients try to connect to the server simultaneously.

Insofar as possible, the effects of various configuration parameters are tested; especially those which generate extra traffic between server to client.

A few specific questions:

  1. Max number of (re)connections per second before server chokes (or, expensiveness of connection establishment)
  2. What happens _when_ server is overloaded?
    • Do the clients connect properly after the server starts responding again? Or do they lose connectivity until they are restarted?
    • Do all the clients end up in expected state if the server is overloaded during connection initialization?
    • Are all client configuration directives (e.g. routes) pushed properly?
  3. Reconnections
    • Are these any different from initial handshakes?

Proposed test cases are shown below:

Test nameVM x OpenVPN instancesAnswer to
C-150x21,2?
C-250x61,2?
C-350x101,2?

The least powerful instances in EC2 (e.g. m1.small) can be slowed down to a crawl fairly easily with 50-100 (real) clients. For more than 100 simultaneous connections several parallel OpenVPN processes are used on the same client instance; although the clients will fail to initialize properly, they should still stress the server in relatively realistic fashion.

Number of simultaneous concurrent connections is increased until server chokes. Each test will be repeated for the following configurations:

IPv4IPv6OpenSSLPolarSSLOpenVPN 2.2.1OpenVPN (master)
X X X
X X X

Question: does IP protocol version or SSL backend affect initial connection establishment significantly?

Test results

Test setup details for m1.small servers

All server instances on EC2 were the same:

  • AMI: OpenVPN Access Server 1.8.3 (ami-31d21358)
  • Operating system: Ubuntu 11.04 "Natty Narwhal" (i386)
  • Kernel: 2.6.38-8-virtual #42-Ubuntu SMP (aki-407d9529)

Client instances on EC2 used a slightly different setup:

  • AMI: Ubuntu 10.04 (ami-81b275e8)
  • Operating system: Ubuntu 10.04 "Lucid Lynx" (i386)
  • Kernel: 2.6.32-317-ec2 #36-Ubuntu SMP (aki-407d9529)

All EC2 instances were created to the same availability zone (us-east-1d). The setup of 100 client instances took ~4 minutes with 34 threads running Fabric (~3 runs per thread).

Test setup details for m1.large servers

All server instances on EC2 were the same:

  • AMI: ami-fd589594
  • Operating system: Ubuntu 11.04 "Natty Narwhal" (amd64)
  • Kernel: 2.6.38-11-virtual #50-Ubuntu SMP (aki-427d952b)

Client instances on EC2 used a slightly different setup:

  • AMI: Ubuntu 10.04 (ami-81b275e8)
  • Operating system: Ubuntu 10.04 "Lucid Lynx" (i386)
  • Kernel: 2.6.32-317-ec2 #36-Ubuntu SMP (aki-407d9529)

All EC2 instances were created to the same availability zone (us-east-1d). The setup of 100 client instances took ~4 minutes with 34 threads running Fabric (~3 runs per thread).

MP-1 results

OpenVPN 2.2.1/m1.small/UDP/IPv4/OpenSSL/5 clients (MP-1)

Test name: 07-mp1-m1.small-openvpn221

Connection establishment

Insignificant CPU load when all clients (5) connected.

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
07-mp1-m1.small-openvpn22110saverage5148.915.563.11
07-mp1-m1.small-openvpn22130saverage5440.415.383.08
07-mp1-m1.small-openvpn22160saverage5888.815.513.1

Server statistics

TestSubtestCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
07-mp1-m1.small-openvpn22110s29.298141.921489.3740
07-mp1-m1.small-openvpn22130s32.969248.395194.44490.060871
07-mp1-m1.small-openvpn22160s35.837548.777497.01250

Analysis

OpenVPN 2.2.1 on an Amazon EC2 m1.small instance can handle 5 OpenVPN client instances well.

MP-2 results

OpenVPN 2.2.1/m1.small/UDP/IPv4/OpenSSL/20 clients (MP-2)

Test name: 13-mp2-m1.small-openvpn221

Connection establishment

Test repeated twice due to timing issues:

  • First run: CPU usage 29-33% for 2 seconds when all clients (20) connected at the same time (at 12:22:02-12:22:03).
  • Second run: CPU usage 100% for 2 seconds...

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
13-mp2-m1.small-openvpn22110saverage20177.2216.530.8265
13-mp2-m1.small-openvpn22130saverage20474.9514.720.736
13-mp2-m1.small-openvpn22160saverage19829.0212.560.661053

Server statistics

TestSubtestReal lengthCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
13-mp2-m1.small-openvpn22110s2124.698954.9197.73550
13-mp2-m1.small-openvpn22130s5626.399154.596596.49540
13-mp2-m1.small-openvpn22160s9826.008156.445497.15540

Analysis

OpenVPN 2.2.1 on an Amazon EC2 m1.small instance seems to be able to handle 20 OpenVPN clients fairly well.

OpenVPN 2.2.1/m1.large/UDP/IPv4/OpenSSL/20 clients (MP-2)

Test names:

  • 16-mp2-m1.large-openvpn221
  • 18-mp2-m1.large-openvpn221
  • 19-mp2-m1.large-openvpn221
  • 20-mp2-m1.large-openvpn221

Connection establishment

During all tests the client initialization phase took 1-2 seconds, judging from server's CPU load and network traffic levels. CPU utilization was 2-6% during this period.

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
16-mp2-m1.large-openvpn22110saverage19386.4237.061.95053
16-mp2-m1.large-openvpn22130saverage20339.859.270.4635
16-mp2-m1.large-openvpn22160saverage201168.6118.910.9455
18-mp2-m1.large-openvpn22110saverage20275.5526.541.327
18-mp2-m1.large-openvpn22130saverage20741.0123.81.19
18-mp2-m1.large-openvpn22160saverage201176.5718.780.939
19-mp2-m1.large-openvpn221120saverage142323.1118.961.35429
20-mp2-m1.large-openvpn221120saverage172369.4219.281.13412

Server statistics

TestSubtestReal lengthCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
16-mp2-m1.large-openvpn22110s228.3484114.97432.16060.0163636
16-mp2-m1.large-openvpn22130s728.8220715.855130.93630.00511111
16-mp2-m1.large-openvpn22160s728.7321415.673631.30020.252833
18-mp2-m1.large-openvpn22110s167.9051215.413234.49070
18-mp2-m1.large-openvpn22130s418.640815.464931.93550.241415
18-mp2-m1.large-openvpn22160s658.6611215.878931.00610.0868
19-mp2-m1.large-openvpn221120s1329.1826415.762231.24030.104886
20-mp2-m1.large-openvpn221120s1329.1555515.873930.82860.0837879

Analysis

A m1.large instance can easily handle 20 clients. As tests were more precisely timed (with "at") than in earlier MP-2 tests (which used chained commands with "sleep" in between), the overly long test periods must have been caused by the iperf server not being able to handle client connections quickly enough.

OpenVPN 2.2.1/m1.large/UDP/IPv4/OpenSSL/20 clients (MP-2)

Test names:

  • 24-mp2.m1.large-openvpn221-1daemon-4iperfs

Connection establishment

Connection establishment took ~1 second, with peak CPU load at 5%.

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
24-mp2.m1.large-openvpn221-1daemon-4iperfs120saverage182181.717.980.998889

Server statistics

TestSubtestReal lengthCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
24-mp2.m1.large-openvpn221-1daemon-4iperfs120s1308.7082315.87429.80970.0595538

MP-3 results

OpenVPN 2.2.1/m1.small/UDP/IPv4/OpenSSL/50 clients (MP-3)

Test name: 14-mp3-m1.small-openvpn221

Connection establishment

Client initialization phase took 10 seconds, judging from server's CPU load and network traffic levels.

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
14-mp3-m1.small-openvpn22110saverage4921019.30.393878
14-mp3-m1.small-openvpn22130saverage49529.215.850.323469
14-mp3-m1.small-openvpn22160saverage481980.6431.650.659375

Server statistics

TestSubtestReal lengthCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
14-mp3-m1.small-openvpn22110s2426.772752.818498.42870
14-mp3-m1.small-openvpn22130+60s30423.582650.421786.85970.0105822

Analysis

Same as for 100 client version of this test, except that the fake performance increase in the last test is not as pronounced.

MP-4 results

OpenVPN 2.2.1/m1.small/UDP/IPv4/OpenSSL/100 clients (MP-4)

Test name: 15-mp4-m1.small-openvpn221

Connection establishment

Client initialization phase took 11 seconds, judging from server's CPU load and network traffic levels.

Client statistics

TestSubtestHostClientsTotal transfer (MB)Total bandwidth (MB/s)Average bandwidth (MB/s)
15-mp4-m1.small-openvpn22110saverage56264.0223.430.418393
15-mp4-m1.small-openvpn22130saverage6259817.170.276935
15-mp4-m1.small-openvpn22160saverage593026.7948.130.815763

Server statistics

TestSubtestReal lengthCPU usr (%)CPU sys (%)CPU total (%)CPU wait (%)
15-mp4-m1.small-openvpn22110s2426.772752.818498.42870
15-mp4-m1.small-openvpn22130+60s30423.582650.421786.85970.0105822

Analysis

100 clients connecting at the same time is clearly above the capabilities of m1.small instance in EC2. Also, the data transfer statistics are skewed because of the long client connection delays, which create a "tail" of clients that raise the average bandwidth numbers because they get a larger proportion of server's resources for their use.

Test environments

General considerations

In environments that are not isolated (e.g. Amazon EC2) the same tests need to be run twice or more to see if external factors affect the results noticeably.

Amazon EC2

Amazon EC2 is a service that can be used to generate lots of VMs, e.g. for performance testing purposes. There are several different types of instance types (=VMs) available. The pricing pricing is divided into hourly rent and bandwidth costs. The latter only apply to traffic going out of the EC2. So, in order to save bandwidth costs, the whole test environment can be built to EC2 within the same availability zone.

Amazon EC2 is used for maximum simultaneous connection testing, where a large number of clients is required. It is also - for now at least - used for maximum throughput tests.

Test environment setup is handled by the Python deployment tool (see below) that utilizes Boto libraries and Fabric under the hood.

Local (LAN)

LAN environment is/can be used for throughput tests. Server and clients are connected to a high-quality, high-bandwidth switch to avoid forming external bottlenecks.

Test environment setup is handled by the Python deployment tool that utilizes Fabric under the hood. Initial installation of client OSes is automated using FAE (Debian) or Kickstart (RHEL/Fedora derivatives).

Configuring test server(s)

Test servers can be/are setup manually or using simple Fabric rules and static host lists, as there are only a few of them.

The required steps:

  • Install a precompiled OpenVPN binary and shared libraries
  • Install an OpenVPN configuration file (for every configuration to test)
  • Start OpenVPN
  • Start memory/CPU usage monitoring tools

For throughput tests also do:

  • Start iperf server instance(s)

Configuring test clients

Configuring test clients consists of the following steps:

  • Install precompiled OpenVPN binary and shared libraries
  • Install OpenVPN configuration file(s)
  • Synchronize the clock using ntpdate: allows properly timed DDOS-style tests
  • Install a test script (modified t_client.rc?) that
    • Verifies that system state is sane
    • Gathers information about system state
    • Launches OpenVPN using a configuration file given as command-line parameter
    • For throughput tests launch iperf client pointed at correct iperf server
    • Wait until the test is over
    • Cleans up system state:
      • Kill OpenVPN gracefully; should it fail, make note of it and use kill -9
      • Flush routes
    • Push collected data into a centralized location (e.g. using SSH)
  • Install crontab entry for each test scenario to launch a test script at a given time

Tools

Deploying clients

Mattock (on IRC) has written a small application that eases deployment of large number of OpenVPN client computers that can adequately stress a server. At the moment (Sep 2011) the app is used for Amazon EC2 deployments, but it's pretty general purpose and easily extendable. It's also not fully functional yet, as one of it components (Fabric) does not play well in this kind of scenarios with updating and/or small modifications. For details, se the included README file.

To checkout this tool (perftest), do the following:

git clone http://build.openvpn.net/tools/perftest.git perftest

The README file contains everything you need to know. At least initially patches should be sent to samuli at openvpn and net, or to mattock who hangs around at #openvpn-devel on the Freenode IRC network (irc.freenode.net).

Generating network traffic

Iperf can be used to generate network traffic. It has previously been used to tune OpenVPN performance: look here.

Collecting data

Dstat can be used to gather detailed resource utilization statistics on the server side.

Notes

Head/tail effect

Initial tests with m1.small server instances, all test phases (connection -> 10s -> 30s -> 60s) were launched from the same (serially executed) shell script, so any delays propagated to later phases. In later tests that used m1.large server instances each subtest was precisely timed using at, so this propagation of delays was eliminated. Regardless of this the delays remained, meaning that a 10 second test could easily become a 20+ second test. The culprit was not the serial nature of timing, but the incapability of the server processes - especially iperf - to establish new connections quickly started to show with as few as 20 clients. This was clearly visible when looking at server-side iperf output: clients connected with delay of several seconds. This had two effects:

  • Prolonging the subtests (e.g. 10 second test could become 20+ second test)
  • Artificially increased the average client bandwidth due to the head and tail of clients that had less competition (bandwidth-vise)

OpenVPN connection initialization seems to be much faster.

This head-tail effect can skew the results badly. These test results are a good example: two identical tests can produce quite different results. There are a few ways to counter this effect:

  • Lengthen the iperf test periods, so that any delays have a less pronounced effect
  • Randomize the client connection to iperf
  • Launch several iperf instances and split client connections between these

These changes combined should produce more reliable results test after test.