Server side testing

OpenVPN server implementations

• OpenVPN 2.x server
  • Linux distros
  • *BSD
  • Windows (yes!)
• Cloudconnexa: https://openvpn.net/cloud-vpn/
  • Developed by OpenVPN Inc.
  • Previously known as OpenVPN Cloud
• Access Server: https://openvpn.net/access-server/
  • Developed by OpenVPN Inc.
  • Based on OpenVPN 2.x
• Azure VPN Gateway: ​https://learn.microsoft.com/en-us/azure/vpn-gateway/vpn-gateway-about-vpngateways
  • Developed by Microsoft
  • Reimplementation of the OpenVPN protocol
  • TCP only
• AWS Client VPN: ​https://docs.aws.amazon.com/vpn/latest/clientvpn-user/client-vpn-user-what-is.html
  • Managed service by AWS
  • Technical details unknown. May be just using OpenVPN 2.x under the hood.
• Softether
  • Open source. Minimal OpenVPN implementation lacks most newer features.

Server-side testing in late 2023

The challenges

• "A 2.6 client on FreeBSD will behave differently than a 2.6 client on Linux with DCO than a 2.6 client on Linux without DCO. and all this might behave different if the server is 2.3 or 2.4." (from cron2 in IRC)
• "Getting proper server instances with arbitrary openvpn versions running is hard, as options have changed quite a bit (so my current server instances won't run on 2.5 due to config incompatibilities - and 2.5 configs won't work with master servers due to certificate shenanigans and such) (from cron2 on IRC)
• "AFAIK DCO on FreeBSD has two drawbacks: 1. Running with UID != 0 breaks it (--user option). 2. "multihome" is not correctly supported. (from mzar on IRC)
• FreeBSD 14 buildbot worker is missing

What is being tested?

• Buildbot launches builds for each "default" build type (openssl, mbedtls) against a set of pre-configured t_client test servers running a version OpenVPN 2.x
  • This includes tests that use HTTP and SOCKS proxies
  • This includes Linux DCO tests
• Daily tests of 2.2, 2.3, 2.4, 2.5 and master clients against "Git" master servers
  • These t_client tests for "testing server functionality" look different from the "test client" - the server provides --client-connect scripts, which can be made to succeed or fail, synchronously or asynchronously, by sending UV_ variables over. This is important to test the server, but for testing clients it would be just a waste of time ("the client does the same thing again and again")

What is not being tested?

• Connecting to arguably important non-FOSS server-side implementations (see above)
• Many client/server combinations
• Servers built from "oldstable" (release/2.5) or "stable" (release/2.6) branches
• Linux DCO on the client or server?

Known issues

• Buildbot can't run P2P tests
• Documentation patches and such will trigger a full buildbot run for no reason

Internal OpenVPN testing at OpenVPN Inc

OpenVPN Inc. has a fairly large client test suite. It consists of OpenVPN clients with a number of different profiles and supporting OpenVPN server side. The test suite did, and maybe still does, run manually. Some of this work could potentially be integrated with community testing efforts and automated.

Current improvement plans

--dev null testing

You can configure OpenVPN to use a "null" device. When you use a "null" device OpenVPN will not attempt to configure network interfaces or routing after starting up (server) or connecting (client). You don't even have to be root or use sudo this way, because no privileged operations are needed. A "--dev null" client can still be used to connect to an OpenVPN server and to verify that the connection was established correctly. Using --dev null allows running multiple, quick client <-> server integration tests on localhost without any dedicated server infrastructure while maintaining very high reliability due to traffic never leaving loopback interfaces.

As the client side is pretty well tested by buildot, the initial goal of --dev null tests is to test the "just compiled openvpn" as a server in various configurations. The clients can be "just compiled openvpn" or any other version that is available locally. The ​current implementation works like this:

1. make check
   1. t_server_null.sh
      1. t_server_null_server.sh
         • Launches the compiled OpenVPN server instances as root (if necessary with sudo) in the background
         • OpenVPN servers exit when all clients have disconnected from all servers
      2. t_server_null_client.sh
         • Launches each individual client with --dev null in the background
         • Each client kills itself after some delay using an "--up" script

The features:

• Parallelized for fairly high performance
• Mostly operating-system agnostic
• Tested on Fedora Linux 38 and FreeBSD 14
• Should be POSIX shell compliant but uses Bash now
• Near 100% reliable (so far no test failures)
• Uses the sample certificates and keys
• Supports running servers directly as root and with sudo
• Supports using different OpenVPN client versions
  • The "current" (just compiled) version
  • Any other OpenVPN versions that is present on the filesystem
• Support testing for success as well as failure
• Test cases (client configurations) and server setups (server configurations) are stored in a configuration file, i.e. data and code have been separated
• Configuration file format is nearly identical to t_client.rc configuration
• Supports a set of default tests, overriding default test settings and adding local tests

Old improvement plans

These plans are probably not going to be implemented as-is. However, so parts of them may end up being used elsewhere.

Brute force?

Using the "brute force" approach would result in a huge combination of configurations to test:

• Linux distro <x>
  • Static OpenVPN 2.5.x client
    • OpenVPN 2.5.x servers
    • OpenVPN 2.6.x servers
    • OpenVPN "Git release/2.6" servers
    • OpenVPN "Git master" servers
  • Static OpenVPN 2.6.x client
    • OpenVPN 2.5 servers
    • OpenVPN 2.6.x servers
    • OpenVPN "Git release/2.6" servers
    • OpenVPN "Git master" servers
  • OpenVPN "Git master" client
    • OpenVPN 2.5 servers
    • OpenVPN 2.6.x servers
    • OpenVPN "Git release/2.6" servers
    • OpenVPN "Git master" servers
  • OpenVPN "Git release/2.5" client
    • OpenVPN 2.5 servers
    • OpenVPN 2.6.x servers
    • OpenVPN "Git release/2.6" servers
    • OpenVPN "Git master" servers
  • OpenVPN "Git release/2.6" client
    • OpenVPN 2.5 servers
    • OpenVPN 2.6.x servers
    • OpenVPN "Git release/2.6" servers
    • OpenVPN "Git master" servers
• Linux distro <y>
  • ...same set of combinations
• FreeBSD version <x>
  • ...same set of combinations
• OpenBSD version <x>
  • ...same set of combinations

This does not even take into account the cases where the OS/distro being used on the client and server is different. So we need to find a reasonable balance between coverage and amount of tests to run.

Implementation of a more thorough server-side testing

This section described one way to implement improved server-side testing using Buildbot and a "farm" of t_client test servers running on different OpenVPN code (release/2.5, release/2.6, master). This has not been tested nor is there a proof of concept of this yet. This may end up being used or not, depending on what the goals are.

Terminology

• t_client server group: a group of t_client servers that run the same version of OpenVPN, for example "master", "release/2.5", "release/2.6" or 2.6.8.
  • OpenVPN server configurations don't need to be 100% identical for "release/2.5", "release/2.6", etc - they just need to be compatible with the configuration of the test clients.
• Message: some mechanism for communicating the state of t_client server groups to buildbot "client" builds

Testing OpenVPN clients built from Git against OpenVPN servers built from Git

Deploying the openvpn executable to applicable t_client test servers

Probably the easiest way to update openvpn "server" executables is by updating them on an NFS share:

Buildbot workers write new openvpn executable to the NFS share and OpenVPN test server instances then execute it.

Ensuring correct test order

Messages that need to be communicated between buildbot "server" and "client" builds:

• t_client server group <x> is ready
• t_client server group <x> has failed

Passing these messages can be achieved in various ways:

• Lock/marker files on an NFS share
• Some form of message broker/event bus/queue (e.g. Redis Streams, Apache Kafka, RabbiMQ)

It is necessary to decouple "server" and "client" builds, _but_ they also must be ordered correctly:

• Normal build steps (e.g. "configure", "make", "make install") can run in parallel for "client" and "server" builds
• "client" builds must not launch t_client tests against a t_client server group before that server group is ready
• In practical terms the "client" builds must have a build step that polls the status of the t_client server group and either wait ("server group not ready yet") or fail ("server group has failed")

Ordering in server builds

t_client server groups

Here's a list of t_client server groups we could have:

• release/2.5
• release/2.6
• release/2.6 + DCO
• master
• master + DCO

Test suites

All t_client server groups will include a set of basic server configurations. In addition later OpenVPN versions will include additional server configuration, for example ones with DCO enabled.

Ordering in client builds

Testing static OpenVPN clients against OpenVPN servers built from Git

The term "static OpenVPN client" means a released version of OpenVPN such as 2.6.8. The build steps would be:

1. Clone OpenVPN from Git
2. Check out the predefined tag
3. Do we want to save CPU and memory?
   1. Yes: use a pre-built executable from NFS
   2. Not really: run the usual build steps
4. Run the usual t_client tests against each t_client server group as they become ready

The number of combination gets quickly out of hand. As server-side setup will be automated that is not a management problem necessarily. However, a large number of combinations slows down testing each commit, so we should probably be a bit conservative initially.

Next steps

We can store "stuff" in ​https://github.com/OpenVPN/openvpn-tests so that other people can get access to it.

1. convert Gert's testing framework to --dev null for testing basic client functionalities
2. implement lightweight client using --dev null (with lwIP). Client should be able to reply to pings and possibly to HTTP requests
3. extend t_client to work with aforementioned mechanism: ping enters the tun and goes to client
4. later: come up with more complex client tests (one client is connected, the next one reconnects multiple times, etc..)
5. later: do we want to test mgmt interface?
6. later later: replace t_client with other test drivers