OpenVPN 2.3.1 consuming tremendous CPU resources on OpenBSD

[mattison]

After having been tasked with setting up a few VPNs for evaluation purposes I noticed that on OpenBSD 5.2 and 5.3, OpenVPN 2.3.1 (running as server) consumes disproportional amounts of CPU resources compared to f.e. FreeBSD and OS X. I have tried multiple ciphers and key lengths, with and without LZO, as well as compiling against both OpenSSL and PolarSSL, but the problem is identical in all scenarios.

Running on one and the same computer with identical server config and cipher + key length etc., OpenVPN consumes around 85-90% of CPU resources on OpenBSD when transiting a mere 15-20 mbit/s worth of data, while on FreeBSD it consumes just around 15% for the same amount of traffic.

Screenshot of top(1) included. If server config file or build configuration is required, please let me know.

[JoshC]

From your top output, most of your CPU is used in system (or kernel) processing time, followed by interrupt processing. Only 15% of the CPU is running userland tasks, which is where openvpn does its work.

So the real question is why your OpenBSD box is doing so much kernel processing compared to your other systems. OpenVPN receives and transmits packets to/from the IP stack both for the tun and physical NICs, so that and firewall processing would be occurring in kernel-space. I'd start hunting there for your CPU issues. Note that the 'CPU' column in top is showing aggregate usage for all types of processing (user + nice + system + interrupt.)

Offhand I'd speculate that checksum offloading, interrupt speed (kernel timer frequency), or firewall processing might be suspect places to look for differences.

[JoshC]

Without any follow-up in 4 weeks, this bug is being marked closed. If you wish to follow-up or re-open the bug, feel free to supply additional relevant details.

[mattison]

Work has been keeping me away. Meanwhile, however, I have been trying the same openvpn config/setup on a different machine running FreeBSD 9.1-release, with no change in behavior. Different OS and different config of the OS, but OpenVPN 2.3.2 is still keeping the machine heavily stalled in terms of system/interrupt in the exact same way as on OpenBSD 5.2/5.3.

[Samuli Seppänen]

If somebody else is having this issue on OpenBSD/FreeBSD, please add a comment.

[Gert Döring]

I've not seen any unduly performance issues on our FreeBSD systems, but I admit that we're only lightly using it - like 20 users, and a few mbit/s throughput.

I think I'll try to get some performance numbers out of these machines, and then see whether this is better or worse than on Linux etc.

Still, significant amounts of CPU spent on "system" hints at "something is going on inside the kernel", and there is not so much OpenVPN could do differently there.

[Samuli Seppänen]

So does this issue appear with latest OpenVPN (Git master) and latest OpenBSD?

[filippobistaffa]

I know this is a very old ticket, but this issue is still present on the latest OpenWrt and Armbian.

[Gert Döring]

Replying to filippobistaffa:

I know this is a very old ticket, but this issue is still present on the latest OpenWrt and Armbian.

Could you be a bit more specific what "this issue" is? What is "Armbian"? Sounds like a Linux variant - while this ticket is about OpenBSD.

Generally speaking, depending on CPU availability and throughput, of course OpenVPN will use all the CPU that's there - that is normal and to be expected Exceptionally high CPU usage with little throughput is what would warrant a ticket... so, numbers please :)

[Gert Döring]

I am closing this ticket now, since it's impossible to deal with "I have the same problem!! (just on a totally different environment)" with no feedback whatsoever.

I have done recent testing with our corp VPN server, which is a FreeBSD 11.3 machine running on a somewhat recent Supermicro hardware, with OpenVPN 2.4.9.

According to prometheus exporter, it will do about 250 Mbit/s with a single OpenVPN process (= using only a single CPU core), and the load on that core will go to about 70-80% then. I consider this "good enough", for better scaling you need a kernel module or multiple parallel OpenVPN processes.

If you see systems with "100% CPU load" at 15-20 Mbit/s, you need to check used ciphers (AES-GCM produces much lower load on systems with AESNI support) - or possibly the system is just not able to do faster (like, a rPI gen 1, with no crypto hw support).

I have no idea why OpenVPN would burn extra CPU on kernel level - what was originally reported - it does not do so on my OpenVPN buildslaves. Maybe something with pf(4) firewalling rules? Or anything unusual on kernel/network level?