**NOTE:** This article is very outdated. Double-check against recent documentation before following the steps outlined here.

It is possible to saturate a 100 Mbps network using an OpenVPN tunnel, achieving throughput very close to that of a regular network interface. However, achieving similar results on gigabit networks and faster is not as straightforward. This page details how to increase the throughput of a VPN tunnel to near-line speed for a 1 Gbps network, with some initial explorations into 10 Gbps networks also discussed.

For this setup, several machines were used, all connected to gigabit switches:

- Two servers running CentOS 5.5 64bit, with an Intel Xeon E5440 CPU at 2.83GHz and an L2 cache size of 6 MB.

- A server running CentOS 5.5 64bit, with an Intel Xeon X5660 CPU at 2.80GHz and an L2 cache size of 12 MB. This CPU supports AES-NI instructions.

- A laptop running Fedora 14 64bit, with an Intel i5-560M CPU at 2.66GHz and an L2 cache size of 3 MB. This CPU also supports AES-NI instructions.

Before starting, the raw network speed was measured using `iperf`, consistently reporting around 940 Mbps, which is almost optimal for a gigabit LAN. The MTU size on all switches in the gigabit LAN was set to 1500.

Understanding how packets flow from the `iperf` client via the OpenVPN tunnel to the `iperf` server is crucial. The following diagram helps to clarify this process:

When an `iperf` packet is sent to the VPN server IP address, it enters the kernel's `tun0` device. The packet is then forwarded to the userspace OpenVPN process, where headers are stripped. The packet is encrypted and signed using OpenSSL calls, configurable via the `--cipher` and `--auth` options.

The packet is then fragmented according to the `--fragment` and `--mssfix` options, and the encrypted packet is sent over the regular network to the OpenVPN server. On the server side, the process is reversed: the packet is reassembled, decrypted, and finally sent out the `tun0` interface.

The default OpenVPN for CentOS 5 is version 2.1.4; the system OpenSSL version is 0.9.8e-fips.

Using a simple shared secret key setup for both server (listener)

openvpn --dev tun --proto udp --port 11000 --secret secret.key --ifconfig 192.168.222.11 192.168.222.10

and client

openvpn --dev tun --proto udp --port 11000 --secret secret.key --ifconfig 192.168.222.10 192.168.222.11 --remote server

an `iperf` result of 156 Mbps is obtained.

Switching to the cipher `aes-256-cbc` results in a performance drop to 126 Mbps. These results were obtained using the two E5440 based servers.

The first tweaks made were:

- Increase the MTU size of the tun adapter (`--tun-mtu`) to 6000 bytes, resembling Jumbo frames on a regular Ethernet LAN. Note that the MTU size on the underlying network switches was not altered.

- Disable OpenVPN's internal fragmentation algorithm using `--fragment 0`.

- Disable OpenVPN's 'TCP Maximum Segment Size' limiter using `--mssfix 0`.

openvpn --dev tun --proto udp --port 11000 --secret secret.key --ifconfig 192.168.222.11 192.168.222.10 --tun-mtu 6000 --fragment 0 --mssfix 0

openvpn --dev tun --proto udp --port 11000 --secret secret.key --ifconfig 192.168.222.10 192.168.222.11 --tun-mtu 6000 --fragment 0 --mssfix 0 --remote server

Now an **iperf** result of **307 Mbps** is obtained.

By playing with the '--tun-mtu' size we obtain (all speeds in Mbps)

| MTU | Blowfish | AES256 |

|-------|----------|--------|

| 1500 | 158 | 126 |

| 6000 | 307 | 220 |

| 9000 | 370 | 249 |

| 12000 | 416 | 252 |

| 24000 | 466 | 259 |

| 36000 | 470 | 244 |

| 48000 | **510** | 247 |

| 60000 | 488 | 221 |

(Please note that for all measurement a standard deviation of ~5% applies)

For the default Blowfish cipher the optimal value for the 'tun-mtu' parameters for a link between these two servers seems to be **48000** bytes. For this 'tun-mtu' setting the VPN throughput increases from 160 Mbps to **510** Mbps.

Similarly, for the AES-256 cipher the optimal value is **24000** bytes.

By increasing the MTU size of the tun adapter **and** by disabling OpenVPN's internal fragmentation routines the throughput can be increased quite dramatically. The reason behind this is that by feeding larger packets to the OpenSSL encryption and decryption routines the performance will go up. The second advantage of not internally fragmenting packets is that this is left to the operating system and to the kernel network device drivers. For a LAN-based setup this can work, but when handling various types of remote users (road warriors, cable modem users, etc) this is not always a possibility.

The default value for **tx_queue_len** in linux is 1000, however, openvpn overrides this default and sets it to 100. You can re-set it to the original default of 1000 by specifying **--txqueuelen 1000**. This can greatly improve throughput in scenarios where using jumbo frames (**--tun-mtu**) is not possible, such as over the internet. The expected performance gain is around 5x, being from 60mbps to almost 300mbps on some tests I carried on a real coast-to-coast scenario (with different ISPs).

| **txqueuelen** | **bandwidth** |

|----------------|---------------|

| 100 | 63 |

| 1000 | 292 |

Please note that, as the name indicates, this only affects the transmission, not the reception. So in case an asymmetric connection is used, this parameter will only have an observable effect on a peer whose transmission speed is greater than ~100mbps.

The second tweak made was to relink OpenVPN 2.1.4 using the OpenSSL 1.0.0a libraries with the Intel AES-NI patch applied. This patch is included by default in Fedora 12 and higher.

Previously it was reported that the Intel AES-NI patch caused the performance on non-AES-NI capable hardware to improve by a factor of 2. Closer investigation showed that the system OpenSSL library 0.9.8e-fips is actually at fault: after recompiling OpenSSL from source, with or without the Intel AES-NI patch, the performance also doubled. The Fedora 12 version of OpenSSL, 1.0.0-fips, and higher do not show this performance penalty.

Testing was done similar to the previous tweak

./openvpn --dev tun --proto udp --port 11000 --secret secret.key --ifconfig 192.168.222.11 192.168.222.10

--tun-mtu 6000 --fragment 0 --mssfix 0 --cipher aes-256-cbc

and client

./openvpn --dev tun --proto udp --port 11000 --secret secret.key --ifconfig 192.168.222.10 192.168.222.11

--tun-mtu 6000 --fragment 0 --mssfix 0 --cipher aes-256-cbc --remote server

Now an **iperf** result of **407 Mbps** is obtained.

By playing with the '--tun-mtu' size we obtain (all speeds in Mbps)

| MTU | Blowfish | AES256 |

|-------|----------|--------|

| 6000 | 310 | 407 |

| 9000 | 385 | 410 |

| 12000 | 417 | 478 |

| 24000 | 470 | 540 |

| 36000 | 510 | 561 |

| 48000 | 500 | **585**|

| 60000 | 500 | 582 |

(Please note that for all measurement a standard deviation of ~5% applies)

For the default Blowfish cipher the optimal value for the 'tun-mtu' parameters for a link between these two servers now seems to be **36000** bytes, although the difference for higher MTU sizes is minimal. Also note that the performance numbers are nearly identical to those generated using the system OpenSSL 0.9.8e-fips library.

When using the AES-256 cipher there is huge performance gain. The optimal MTU value now is **48000** bytes, but overall performance increased by a factor of 2 for nearly all MTU sizes.

Compiling OpenSSL from scratch doubles the OpenSSL speed by a factor of 2. This can also be seen when running

This is caused mainly by the fact that the CentOS-supplied OpenSSL 0.9.8e-fips library seems broken. The Fedora 12+ supplied OpenSSL 1.0.0-fips library does not induce the same penalty.

The real performance gain is expected from the AES-NI capable hardware, such as the Intel Xeon X5660 and i5-560M CPUs.

Using the exact same setup as above, but now using

and by using

for the server and

for the client we obtain the following results (all speeds in Mbps):

| | AES128 | AES256 |

|---------------|--------|--------|

| X5660 -> i5-560 | 885 | 878 |

| i5-560 -> X5660 | 748 | 543 |

(Please note that for all measurement a standard deviation of ~5% applies)

Some initial conclusions:

* the AES-NI capable server CPU clearly shows a huge performance gain: it is nearly capable of saturating the gigabit network.

For reference, the above test was also run with encryption and signing disabled:

openvpn --dev tun --proto udp --port 11000 --secret secret.key --ifconfig 192.168.222.11 192.168.222.10 --tun-mtu 9000 --fragment 0 --mssfix 0 --cipher none --auth none

Now an **iperf** result of **930 Mbps** is obtained, which shows that performance is not so much limited by the division between kernel-space and user-space processes, but mostly by the encryption and decryption routines as found in the OpenSSL libraries.

The "plaintext" test (i.e., encryption and signing disabled) was repeated on two machines connected to a 10 Gbps switch.

Again, all results were obtained using `iperf`.

| Mode | MTU | Speed |

|------------------------------|-------|-------------|

| "raw" | 1500 | 8.8 Gbps |

| "raw", no cksum offloading | 1500 | 3.8 Gbps |

| via tun | 60000 | 3.6 Gbps |

| via tun | 48000 | 2.4 Gbps |

| via tun | 9000 | 1.3 Gbps |

This shows the limits of the kernel-space/user-space division that is present in Linux. It also shows how important TCP offloading is: if the TCP checksums need to be calculated in kernel space the performance drops by 50%!

The performance of OpenVPN over this network is not yet understood, as the raw OpenSSL speed of the computers in this network is much lower than that of servers listed above. Unfortunately, it was not possible to use two AES-NI capable PCs on this network.