wiki:ServerSideTestingImprovementPlan

Server side testing

OpenVPN server implementations

Server-side testing in late 2023

The challenges

  • "A 2.6 client on FreeBSD will behave differently than a 2.6 client on Linux with DCO than a 2.6 client on Linux without DCO. and all this might behave different if the server is 2.3 or 2.4." (from cron2 in IRC)
  • "Getting proper server instances with arbitrary openvpn versions running is hard, as options have changed quite a bit (so my current server instances won't run on 2.5 due to config incompatibilities - and 2.5 configs won't work with master servers due to certificate shenanigans and such) (from cron2 on IRC)
  • "AFAIK DCO on FreeBSD has two drawbacks: 1. Running with UID != 0 breaks it (--user option). 2. "multihome" is not correctly supported. (from mzar on IRC)
  • FreeBSD 14 buildbot worker is missing

Brute force?

Using the "brute force" approach would result in a huge combination of configurations to test:

  • Linux distro <x>
    • Static OpenVPN 2.5.x client
      • OpenVPN 2.5.x servers
      • OpenVPN 2.6.x servers
      • OpenVPN "Git release/2.6" servers
      • OpenVPN "Git master" servers
    • Static OpenVPN 2.6.x client
      • OpenVPN 2.5 servers
      • OpenVPN 2.6.x servers
      • OpenVPN "Git release/2.6" servers
      • OpenVPN "Git master" servers
    • OpenVPN "Git master" client
      • OpenVPN 2.5 servers
      • OpenVPN 2.6.x servers
      • OpenVPN "Git release/2.6" servers
      • OpenVPN "Git master" servers
    • OpenVPN "Git release/2.5" client
      • OpenVPN 2.5 servers
      • OpenVPN 2.6.x servers
      • OpenVPN "Git release/2.6" servers
      • OpenVPN "Git master" servers
    • OpenVPN "Git release/2.6" client
      • OpenVPN 2.5 servers
      • OpenVPN 2.6.x servers
      • OpenVPN "Git release/2.6" servers
      • OpenVPN "Git master" servers
  • Linux distro <y>
    • ...same set of combinations
  • FreeBSD version <x>
    • ...same set of combinations
  • OpenBSD version <x>
    • ...same set of combinations

This does not even take into account the cases where the OS/distro being used on the client and server is different. So we need to find a reasonable balance between coverage and amount of tests to run.

What is being tested?

  • Buildbot launches builds for each "default" build type (openssl, mbedtls) against a set of pre-configured t_client test servers running a version OpenVPN 2.x
    • This includes tests that use HTTP and SOCKS proxies
    • This includes Linux DCO tests
  • Daily tests of 2.2, 2.3, 2.4, 2.5 and master clients against "Git" master servers
    • These t_client tests for "testing server functionality" look different from the "test client" - the server provides --client-connect scripts, which can be made to succeed or fail, synchronously or asynchronously, by sending UV_ variables over. This is important to test the server, but for testing clients it would be just a waste of time ("the client does the same thing again and again")

What is not being tested?

  • Connecting to arguably important non-FOSS server-side implementations (see above)
  • Many client/server combinations
  • Servers built from "oldstable" (release/2.5) or "stable" (release/2.6) branches
  • Linux DCO on the client or server?

Known issues

  • Buildbot can't run P2P tests
  • Documentation patches and such will trigger a full buildbot run for no reason

Internal OpenVPN testing at OpenVPN Inc

OpenVPN Inc. has a fairly large client test suite. It consists of OpenVPN clients with a number of different profiles and supporting OpenVPN server side. The test suite did, and maybe still does, run manually. Some of this work could potentially be integrated with community testing efforts and automated.

Potential improvements for 2024

The priority of improvements might be the following - please create/update/remove based on community discussions:

  1. Testing Git master clients against release/2.5 and release/2.6 servers?
  2. Testing against DCO-enabled servers?
  3. Testing with DCO-enabled clients?

Implementation of a more thorough server-side testing

This section described one way to implement improved server-side testing using Buildbot and a "farm" of t_client test servers running on different OpenVPN code (release/2.5, release/2.6, master). This has not been tested nor is there a proof of concept of this yet. This may end up being used or not, depending on what the goals are.

Terminology

  • t_client server group: a group of t_client servers that run the same version of OpenVPN, for example "master", "release/2.5", "release/2.6" or 2.6.8.
    • OpenVPN server configurations don't need to be 100% identical for "release/2.5", "release/2.6", etc - they just need to be compatible with the configuration of the test clients.
  • Message: some mechanism for communicating the state of t_client server groups to buildbot "client" builds

Testing OpenVPN clients built from Git against OpenVPN servers built from Git

Deploying the openvpn executable to applicable t_client test servers

Probably the easiest way to update openvpn "server" executables is by updating them on an NFS share:

Buildbot workers write new openvpn executable to the NFS share and OpenVPN test server instances then execute it.

Ensuring correct test order

Messages that need to be communicated between buildbot "server" and "client" builds:

  • t_client server group <x> is ready
  • t_client server group <x> has failed

Passing these messages can be achieved in various ways:

  • Lock/marker files on an NFS share
  • Some form of message broker/event bus/queue (e.g. Redis Streams, Apache Kafka, RabbiMQ)

It is necessary to decouple "server" and "client" builds, _but_ they also must be ordered correctly:

  • Normal build steps (e.g. "configure", "make", "make install") can run in parallel for "client" and "server" builds
  • "client" builds must not launch t_client tests against a t_client server group before that server group is ready
  • In practical terms the "client" builds must have a build step that polls the status of the t_client server group and either wait ("server group not ready yet") or fail ("server group has failed")

Ordering in server builds

t_client server groups

Here's a list of t_client server groups we could have:

  • release/2.5
  • release/2.6
  • release/2.6 + DCO
  • master
  • master + DCO

Test suites

All t_client server groups will include a set of basic server configurations. In addition later OpenVPN versions will include additional server configuration, for example ones with DCO enabled.

Ordering in client builds

Testing static OpenVPN clients against OpenVPN servers built from Git

The term "static OpenVPN client" means a released version of OpenVPN such as 2.6.8. The build steps would be:

  1. Clone OpenVPN from Git
  2. Check out the predefined tag
  3. Do we want to save CPU and memory?
    1. Yes: use a pre-built executable from NFS
    2. Not really: run the usual build steps
  4. Run the usual t_client tests against each t_client server group as they become ready

The number of combination gets quickly out of hand. As server-side setup will be automated that is not a management problem necessarily. However, a large number of combinations slows down testing each commit, so we should probably be a bit conservative initially.

Next steps

We can store "stuff" in https://github.com/OpenVPN/openvpn-tests so that other people can get access to it.

  1. convert Gert's testing framework to --dev null for testing basic client functionalities
  2. implement lightweight client using --dev null (with lwIP). Client should be able to reply to pings and possibly to HTTP requests
  3. extend t_client to work with aforementioned mechanism: ping enters the tun and goes to client
  4. later: come up with more complex client tests (one client is connected, the next one reconnects multiple times, etc..)
  5. later: do we want to test mgmt interface?
  6. later later: replace t_client with other test drivers
Last modified 7 days ago Last modified on 02/27/24 14:41:55

Attachments (3)

Download all attachments as: .zip