wiki:PerformanceTesting

Version 18 (modified by Samuli Seppänen, 12 years ago) (diff)

--

Introduction

There have been significant changes in the future 2.3 branch since the relatively stable and conservative 2.1.x and 2.2.x releases. Therefore there's a definite need for wide testing in various areas, including performance. This page tries to outline what the most relevant tests are so that they can be ran on a regular basis during development to spot any regressions and bottlenecks. These tests are primarily aimed at the future OpenVPN 2.3 branch (current "master"). However, same setup can used to test older versions, too. Focus is on testing the most common and/or potentially useful configurations, as testing all combinations is practically impossible.

Goals

The tests have several goals:

  • Get an idea of OpenVPN performance with any given parameters
  • Identify performance bottlenecks and fix the underlying issues insofar as possible
  • Identify non-linear resource comsumption growth, e.g. find out if 100 clients consumes (significantly) more resources than that of 10 clients (multiplied by ten)
  • Detect and fix abnormal behavior (e.g. partially configured clients) during peak load periods

When these tests are repeated over time, they also allow spotting regressions.

Prequisites

Identification of performance bottlenecks requires plotting resource consumption, e.g. memory usage and processor load. This, in turn may require selective adding of new debugging code, so that excessively verbose, generic logging does not degrade performance and thus produce biased results. Maximum performance tests don't have this requirement, as they only need minimal logging. In addition, spotting and debugging configuration errors requires collecting information about OpenVPN's state on the client side.

There are further requirements, depending on the test environments being used; see below for details.

Test cases

Here we should have a high-level list of test cases. Technical details (e.g. OpenVPN client configuration files) can also be attached. Mattock can then convert these descriptions into Fabric code which makes the magic happen.

Throughput tests

There are a number of factors affecting throughput:

  • number of clients
  • bandwidth per client
  • encryption algorithms/ciphers
  • compression
  • packet size
  • transport protocol (TCP/UDP)
  • network errors
  • others factors, which?

There are a few specific questions that need an answer (feel free to add more):

  1. Maximum throughput using various parameters
  2. How does (excessive) packet loss affect performance?
  3. Does number of clients affect performance, if data transfer rate remains the same? (or "non-linear resource consumption growth")

Below is a list of proposed test cases. All bandwidth (BW) numbers are in MB/s (megabytes per second), as measured by iperf.

Test nameMin clientsMax clientsBW/clientBandwidth riseDescriptionAnswer to
TP-1110100GradualFew clients with high bandwidth1,3
TP-21010010GradualMany clients with medium bandwidth1,3
TP-31010001GradualHorde of clients with low bandwidth1,3

Protocol needs to be UDP, as TCP (in iperf) prevents controlling throughput. Each test will be repeated for each of the following combinations:

UDPIPv4IPv6OpenSSLPolarSSLOpenVPN 2.2.1OpenVPN (master)
XX X X
XX X X
X XX X
X X X X
XX X

Question: Should a subset of these tests repeated for TCP?

Connection tests

Connection tests are aimed at identifying bottlenecks during spikes of (re)connection initiations, e.g. when 100 clients try to connect to the server simultaneously.

Insofar as possible, the effects of various configuration parameters are tested; especially those which generate extra traffic between server to client.

A few specific questions:

  1. Max number of (re)connections per second before server chokes (or, expensiveness of connection establishment)
  2. What happens _when_ server is overloaded?
    • Do the clients connect properly after the server starts responding again? Or do they lose connectivity until they are restarted?
    • Do all the clients end up in expected state if the server is overloaded during connection initialization?
    • Are all client configuration directives (e.g. routes) pushed properly?
  3. Reconnections
    • Are these any different from initial handshakes?

Proposed test cases are shown below:

Test nameMax clientsDescriptionAnswer to
C-110Simultaneous connection initiation1
C-250Simultaneous connection initiation1,2?
C-3100Simultaneous connection initiation1,2?

Number of simultaneous concurrent connections is increased until server chokes. Each test will be repeated for the following configurations:

IPv4IPv6OpenSSLPolarSSLOpenVPN 2.2.1OpenVPN (master)
X X X
X X X

Question: does IP protocol version or SSL backend affect initial connection establishment significantly?

Test results

Test setup details

All server instances on EC2 were the same:

  • AMI: OpenVPN Access Server 1.8.3 (ami-31d21358)
  • Operating system: Ubuntu 11.04 "Natty Narwhal" (i386)
  • Kernel: 2.6.38-8-virtual #42-Ubuntu SMP (aki-407d9529)

Client instances on EC2 used a slightly different setup:

  • AMI: Ubuntu 10.04 (ami-81b275e8)
  • Operating system: Ubuntu 10.04 "Lucid Lynx" (i386)
  • Kernel: 2.6.32-317-ec2 #36-Ubuntu SMP (aki-407d9529)

All EC2 instances were created to the same availability zone (us-east-1d).

Test environments

General considerations

In environments that are not isolated (e.g. Amazon EC2) the same tests need to be run twice or more to see if external factors affect the results noticeably.

Amazon EC2

Amazon EC2 is a service that can be used to generate lots of VMs, e.g. for performance testing purposes. There are several different types of instance types (=VMs) available. The pricing pricing is divided into hourly rent and bandwidth costs. The latter only apply to traffic going out of the EC2. So, in order to save bandwidth costs, the whole test environment can be built to EC2 within the same availability zone.

Amazon EC2 is used for maximum simultaneous connection testing, where a large number of clients is required. It is also - for now at least - used for maximum throughput tests.

Test environment setup is handled by the Python deployment tool (see below) that utilizes Boto libraries and Fabric under the hood.

Local (LAN)

LAN environment is/can be used for throughput tests. Server and clients are connected to a high-quality, high-bandwidth switch to avoid forming external bottlenecks.

Test environment setup is handled by the Python deployment tool that utilizes Fabric under the hood. Initial installation of client OSes is automated using FAE (Debian) or Kickstart (RHEL/Fedora derivatives).

Configuring test server(s)

Test servers can be/are setup manually or using simple Fabric rules and static host lists, as there are only a few of them.

The required steps:

  • Install a precompiled OpenVPN binary and shared libraries
  • Install an OpenVPN configuration file (for every configuration to test)
  • Start OpenVPN
  • Start memory/CPU usage monitoring tools

For throughput tests also do:

  • Start iperf server instance(s)

Configuring test clients

Configuring test clients consists of the following steps:

  • Install precompiled OpenVPN binary and shared libraries
  • Install OpenVPN configuration file(s)
  • Synchronize the clock using ntpdate: allows properly timed DDOS-style tests
  • Install a test script (modified t_client.rc?) that
    • Verifies that system state is sane
    • Gathers information about system state
    • Launches OpenVPN using a configuration file given as command-line parameter
    • For throughput tests launch iperf client pointed at correct iperf server
    • Wait until the test is over
    • Cleans up system state:
      • Kill OpenVPN gracefully; should it fail, make note of it and use kill -9
      • Flush routes
    • Push collected data into a centralized location (e.g. using SSH)
  • Install crontab entry for each test scenario to launch a test script at a given time

Tools

Deploying clients

Mattock (on IRC) has written a small application that eases deployment of large number of OpenVPN client computers that can adequately stress a server. At the moment (Sep 2011) the app is used for Amazon EC2 deployments, but it's pretty general purpose and easily extendable. It's also not fully functional yet, as one of it components (Fabric) does not play well in this kind of scenarios with updating and/or small modifications. For details, se the included README file.

To checkout this tool (perftest), do the following:

git clone http://build.openvpn.net/tools/perftest.git perftest

The README file contains everything you need to know. At least initially patches should be sent to samuli at openvpn and net, or to mattock who hangs around at #openvpn-devel on the Freenode IRC network (irc.freenode.net).

Generating network traffic

Iperf can be used to generate network traffic. It has previously been used to tune OpenVPN performance: look here.

Collecting data

Dstat can be used to gather detailed resource utilization statistics on the server side.