wiki:ServerSideTestingImprovementPlan

Server side testing

OpenVPN server implementations

Server-side testing in July 2024

The challenges

  • "A 2.6 client on FreeBSD will behave differently than a 2.6 client on Linux with DCO than a 2.6 client on Linux without DCO. and all this might behave different if the server is 2.3 or 2.4." (from cron2 in IRC)
  • "Getting proper server instances with arbitrary openvpn versions running is hard, as options have changed quite a bit (so my current server instances won't run on 2.5 due to config incompatibilities - and 2.5 configs won't work with master servers due to certificate shenanigans and such) (from cron2 on IRC)
  • "AFAIK DCO on FreeBSD has two drawbacks: 1. Running with UID != 0 breaks it (--user option). 2. "multihome" is not correctly supported. (from mzar on IRC)
  • FreeBSD 14 buildbot worker is missing

What is being tested?

  • Client tests (t_client.sh)
    • Buildbot launches builds for each "default" build type (openssl, mbedtls) against a set of pre-configured t_client test servers running a version OpenVPN 2.x
      • This includes tests that use HTTP and SOCKS proxies
      • This includes Linux DCO tests
  • Server tests (t_server_null.sh)
    • Buildbot launches multiple server instances and launches multiple clients instances against the servers
    • Both servers and clients can have arbitrary configurations
    • Detailed description of t_server_null.sh below
  • Daily scheduled t_client.sh tests of 2.2, 2.3, 2.4, 2.5 and master clients against "Git" master servers
    • These t_client tests for "testing server functionality" look different from the "test client" - the server provides --client-connect scripts, which can be made to succeed or fail, synchronously or asynchronously, by sending UV_ variables over. This is important to test the server, but for testing clients it would be just a waste of time ("the client does the same thing again and again")

What is not being tested?

  • Connecting to some arguably important non-FOSS / non-standard server-side implementations (see above)
  • Connecting to "just compiled" Git "master" server using "oldstable" (release/2.5) or "stable" (release/2.6) clients
  • Many other client/server combinations
  • Servers built from "oldstable" (release/2.5) or "stable" (release/2.6) branches
  • Linux DCO on the client or server?

Known issues

  • Buildbot can't run P2P tests

Internal OpenVPN testing at OpenVPN Inc

OpenVPN Inc. has a fairly large client test suite. It consists of OpenVPN clients with a number of different profiles and supporting OpenVPN server side. The test suite did, and maybe still does, run manually. Some of this work could potentially be integrated with community testing efforts and automated.

Next steps

We can store "stuff" in https://github.com/OpenVPN/openvpn-tests so that other people can get access to it.

  1. t_server_null.sh: test older (2.5, 2.6) clients against the "just compiled" Git "master" server
  2. add ARM64-based buildbot workers to improve server (and client) coverage
  3. convert Gert's testing framework to --dev null for testing basic client functionalities
  4. implement lightweight client using --dev null (with lwIP). Client should be able to reply to pings and possibly to HTTP requests
  5. extend t_client to work with aforementioned mechanism: ping enters the tun and goes to client
  6. later: come up with more complex client tests (one client is connected, the next one reconnects multiple times, etc..)
  7. later: do we want to test mgmt interface?
  8. later later: replace t_client with other test drivers

t_server_null.sh

You can configure OpenVPN to use a "null" device. When you use a "null" device OpenVPN will not attempt to configure network interfaces or routing after starting up (server) or connecting (client). You don't even have to be root or use sudo this way, because no privileged operations are needed. A "--dev null" client can still be used to connect to an OpenVPN server and to verify that the connection was established correctly. Using "--dev null" allows running multiple, quick client <-> server integration tests on localhost without any dedicated server infrastructure while maintaining very high reliability due to traffic never leaving the loopback interface.

The tests run the "just compiled openvpn" as a server in various configurations. The clients are currently "just compiled openvpn" as well, but they can be any other version that is available locally. The current implementation works like this:

  1. make check
    1. t_server_null.sh
      1. t_server_null_server.sh
        • Launches the compiled OpenVPN server instances as root (if necessary with sudo or doas) in the background
        • OpenVPN servers exit when all clients have disconnected from all servers
      2. t_server_null_client.sh
        • Launches each individual client with --dev null in the background
        • Each client kills itself after some delay using an "--up" script

The features:

  • Parallelized for fairly high performance
  • Operating system agnostic
  • Tested on the following platforms:
    • Fedora Linux 38-40 and "rawhide"
    • Ubuntu 20.04, 22.04 and 24.04
    • Debian 10-12 and "unstable"
    • OpenSuSE Leap 15
    • Alpine Linux
    • Arch Linux
    • FreeBSD 12-14
    • OpenBSD 6.8 and 7.5
    • NetBSD 8.1 and 10.0
  • POSIX shell compliant
  • Tested with the following shells:
    • Bash
    • Dash
    • Yash
    • Ksh
    • FreeBSD "/bin/sh" (a bourne shell variant)
    • NetBSD "/bin/sh" (a bourne shell variant)
  • Near 100% reliable (so far no test failures)
  • Uses the sample certificates and keys
  • Supports running servers directly as root and with sudo/doas
  • Supports using different OpenVPN client versions
    • The "current" (just compiled) version
    • Any other OpenVPN versions that is present on the filesystem
  • Support testing for success as well as failure
  • Test cases (client configurations) and server setups (server configurations) are stored in a configuration file, i.e. data and code have been separated
  • Configuration file format is nearly identical to t_client.rc configuration
  • Supports a set of default tests, overriding default test settings and adding local tests
Last modified 3 weeks ago Last modified on 07/04/24 13:43:17

Attachments (3)

Download all attachments as: .zip