This page outlines the efforts taken to maintain OpenVPN's code quality without excessive compromises on development speed.

Here's an overview of our current, work in progress and proposed QA tools/processes for OpenVPN 2.x:

|Tool/process |Automated |In use |Scope |Extra requirements |B.patch|B.merge|A.merge|B.release|B.distro merge|

|- |- |- |- |- |- |- |- |- |- |

|"make check" |No[5] |Yes |Build |None |X | | | | |

|vagrant (build) |No[6] |Yes |Build |Vagrant installed |X | | | | |

|vagrant (server) |No[7] |Needs work |Integration |Vagrant installed |X | | | | |

|unit tests |Yes |Yes |Regression |Cmocka installed |X | | | | |

|code review |No |Yes |Everything |None | |X | | | |

|GitHub Actions |Yes |Yes |Build |Use a GitHub PR | |X | | | |

|appveyor |Yes |Yes |Build |Use a GitHub PR | |X | | | |

|patchwork |Yes |Yes |Helps code review |None | |X | | | |

|push scripts |No |Yes |? |Scripts installed | |X | | | |

|buildbot |Yes |Yes |Build,integration |None | |(X) [3]|X | | |

|win builds![1] |Yes |Yes |Build |None | |(X) [3]|X | | |

|GHA win snapshots![1]|Yes |Yes |Everything |None | | | |X | |

|win testsuite![2] |No[4] |Yes |Integration |Windows | |(X) [3]|(X) [3]|X | |

|linux packages |No[4] |Yes |Everything |Debian OS | |(X) [3]|(X) [3]|(X) [3] |X |

|distro qa![8] |- |- |- |None | | | | |X |

Most OpenVPN subprojects, including [openvpn](https://github.com/OpenVPN/openvpn), [openvpn-build](https://github.com/OpenVPN/openvpn-build), [openvpn-gui](https://github.com/OpenVPN/openvpn-gui) and [tap-windows6](https://github.com/OpenVPN/tap-windows6) now have Travis CI and/or !AppVeyor support.

Notes:

1. https://github.com/mattock/openvpn-windows-buildtest

1. https://github.com/mattock/openvpn-windows-test

1. This tool/process could be adapted to catch errors earlier, at this phase

1. Could be automated, but requires a fair amount of work; integration with Windows snapshots could be accomplished with [Chocolatey](https://chocolatey.org/).

1. "make check" as such does not run automatically at build. However, some of the other CI tools (travis, buildbot) do run it on every commit.

1. Vagrant VMs allow any developer - core or otherwise - to test OpenVPN on any esoteric platform before sending a patch. The process is manual right now.

1. OpenVPN servers provided as Vagrant VMs can be used to launch client connectivity tests against OpenVPN servers configured in various ways. This is akin to what Buildbot does at the moment, but the process is/will be more manual and oriented towards single developers.

1. The "Distro QA" line is here to emphasize the fact that downstread distributors of OpenVPN have their own QA processes in place. A large part of OpenVPN downloads come through these distributors, so we indirectly benefit from their QA also.

Key:

* B.patch = (Typically catches errors) before a patch is published

* B.merge = before merging the patch (to Git "master")

* A.merge = after merging the patch (to Git "master")

* B.release = before making a release

* B.distro merge = before distributions merge the release to their repositories/ports

*Scope* is very high-level here by design. We can improve our QA in two main ways:

1. Catch the problems earlier

1. Make each tool/process catch more problems

This overview was created originally for community meeting held on [7th November 2016](/Topics-2016-11-07).

Static testing usually refers to [static code analysis](https://en.wikipedia.org/wiki/Static_code_analysis), which is baked in into our development process in the form of mandatory ACK process which every patch has to go through. The ACK process not only improves code quality, it also prevents highly specialized or rarely used features from polluting the codebase. Code reviews are especially important for patches coming from non-core contributors who may not be familiar with OpenVPN's coding practices. That said, the review constantly catch small issues in patches sent by the core developers also.

OpenVPN's codebase is scanned using [Coverity Scan](http://scan.coverity.com) periodically. This detects many potential security vulnerabilities.

Dynamic black-box testing means trying out an application and verifying if it works as intended. In closed-source software development which is organized around a [waterfall model](http://en.wikipedia.org/wiki/Waterfall_model) there are usually dedicated testers who do various scripted or intuitive/exploratory tests to verify that an application works as intended. The testing usually happens just before launch. In complex applications (such as OpenVPN) testing even a small fraction of functionality would be impractical and very costly. Fortunately, in [Lean software development](http://en.wikipedia.org/wiki/Lean_software_development) methodologies such as [Scrum](http://en.wikipedia.org/wiki/Scrum_%28development%29) and especially in community-driven OSS development doing *extensive* dedicated testing is in general just a waste of time. It is replaced by

* Constant quality assurance achieved with static whitebox technique (e.g. code reviews)

* Testing in real environments (by users)

This said, a small amount of dedicated, dynamic testing (a.k.a. [smoke testing](http://en.wikipedia.org/wiki/Smoke_testing)) goes into each release to catch the most obvious errors. All new features are tested separately before a patch is accepted to Git.

If you want to help with pre-release testing please don't hesitate to [contact the developers](../Pages/GettingHelp) about it.

In the past performance tests have been conducted to measure OpenVPN performance:

* [Performance testing wiki page](../Archived/PerformanceTesting)

We don't currently have a reliable test network which we could use to detect small performance regressions. Please don't hesitate to [contact us](/GettingHelp) if you think you could help us out with this problem.

In OpenVPN (and most other open source projects), the stability of stable releases is ensured with *real-life testing* by it's users during all phases of software development, starting from patches sent to the mailing list, followed by development code in Git and leading into stable releases. There are at least two kinds of *barriers* to using pre-release code:

* Risk avoidance

* Unfamiliarity with required tools (e.g. Git)

* Difficulty of deployment, e.g. building software from sources (especially on Windows)

This means that, the closer we get to release, the more people we can expect to be testing the codebase. The figures below are not based on any real data and can only be considered rough estimates:

|**Git**|**Snapshots**|**Alpha**|**Beta**|**RC**|**Release**|

|- |- |- |- |- |- |

|0.1%|0.2%|1%|5%|10%|99%|

The use of snapshots help overcome some of the technical barriers. The only way to overcome psychological barriers is to speed up the release cycle. This results in new features get into wide circulation faster, which in turn results into issues being reported more quickly. This also gives more confidence in integrity of stable releases. On the flipside, more bugs will probably end up in the initial versions of the stable releases, which may create further disincentives for cautious users to install initial release versions of OpenVPN.

We currently provide Windows builds for each commit to the release branch(es) and the master branch:

* http://build.openvpn.net/downloads/snapshots/

Each installer has a timestamp which determines how new it is. Please include the full name of the installer you've used when reporting problems.

The first line of defense is the GitHub Actions integration in GitHub, which ensures that pull requests (where allowed) do not break the main codebase badly, and that build failures caused by Git push are noticed very quickly. How GitHub Actions is used depends on the OpenVPN subproject in question.

The project also has a Buildbot CI/CD system, which drives several buildbot workers. These together form a [continuous integration](http://en.wikipedia.org/wiki/Continuous_integration ) environment for OpenVPN. Each of the workers is running a different operating system, and every commit to the Git repository triggers a build on each. In addition each openvpn binary built with default configure options with either OpenSSL or mbedTLS makes connections to several test servers using several different configurations. This ensures that

* OpenVPN projects build properly on a variety of platforms

* The very basic functionality is not horribly broken

Trusted OpenVPN core developers have access to Buildbot and the Git repository it uses, which allows extensively testing feature branches for example, before sending a patch or creating a PR.

Buildbot currently (September 2022) does CI/CD for the following projects:

* openvpn

* openvpn3

* openvpn3-linux

* ovpn-dco

For further details see the SettingUpBuildslave article.

The OpenVPN project uses the [CMocka unit testing framework](https://cmocka.org/). If you have experience with unit testing and want to help us spot regressions, please [contact us](../Pages/GettingHelp).

Windows is different enough from the *NIX platforms to require tailored testing procedures. Those are described in detail on the WindowsTesting Wiki page.

In addition to the above there are a bunch of tools being used by different people for testing. It would benefit everyone if some of these could be integrated with the existing QA procedures in an automated way:

* dazo has some test OpenVPN config files which he uses for testing

* ordex has a test setup for quickly testing ovpn-dco

* there are a bunch of OpenVPN servers and matching client profiles primarily used for testing Access Server releases, possibly also for OpenVPN Cloud

In particular it would be nice to be able to do automated testing against servers other than the t_client servers. Whether this means extending t_client or having a parallel system remains to be seen after the requirements have been gathered.