OpenVPN project uses [Buildbot](https://buildbot.net) to help increase code quality. Buildbot is a Python application that can work in either *master* or *worker* mode. The *buildmaster* is the core server which accepts connections from *workers* and tells them what they should do. Typically the clients fetch latest sources and reports any build problems to buildbot which in turn informs developers via email. In software engineering this is called [http://en.wikipedia.org/wiki/Continuous_integration Continous integration] and helps prevent build problems go unnoticed for extended time periods. The clients (workers) can and should run on a variety of hardware / OS platforms. For the server (buildmaster) the OS choice is largely irrelevant. Buildbot is described in more detail in the [Buildbot manual](https://docs.buildbot.net/current/manual/index.html).

As the number of workers can easily get out of hand, the OpenVPN project can make use of *your* help. If you're interested in donating a buildslave please contact the buildmaster admins:

* Gert (cron2 on IRC)

* Samuli (mattock on IRC)

**NOTE:** The workers need root access to be able to connect to the t_client.sh test servers using OpenVPN. Moreover, anyone who controls the buildmaster can do whatever he wants on the workers by simply modifying the build procedure. For this reason you should only run workers on expendable virtual machines or containers. That said, you can grant limited *sudo* privileges to a normal user to run buildbot as a normal user: see section "Running builtdot as non-root user" for details.

Currently buildbot builds the following projects:

* openvpn

* openvpn3

* openvpn3-linux

* ovpn-dco

You can decide which ones you want your buildbot worker to build.

Here's a comprehensive list of workers already running (as of June 2022):

* debian-10

* debian-11

* debian-unstable

* arch

* fedora-34

* opensuse-leap-15

* ubuntu-1804

* ubuntu-2004

* ubuntu-2110

* cron2-fbsd74

* cron2-fbsd11

* cron2-fbsd12

* cron2-fbsd13

* cron2-nbsd81

* cron2-obsd68

* cron2-oi2019

A Linux VM will probably need about 1024MB of memory.The VM should have at least 8GB of diskspace, but adding a bit more helps avoid problems later on.

The first step is to ensure you can build all the projects you wish to build. If manually building does not work, it cannot work in buildbot, either. Please follow the instructions for each codebase on how to build.

Our buildmaster is accessible only via the community VPN. Please ask *mattock* on IRC for VPN config for your worker. You will get the config in Signal or in an GnuPG-encrypted email. The community VPN server pushes DNS servers as well, so you may want to process those if you want to use DNS names instead of IP addresses.

A simple ping test should be enough to verify that your worker can reach buildmaster:

If both work, it means your VPN client is properly configured. If pinging by name fails, but by IP works, then your VPN client is not processing DNS settings pushed by the OpenVPN server properly. But you can still point your worker to the IP, so it is not a blocker.

Note that Buildbot no longer supports Python 2.x, so ensure that your to-be worker has a working Python 3 installation.

A simplistic procedure for Ubuntu 20.04 is described below. As t_client tests need root-level privileges anyways, here we show how to run a worker as root:

You can get the worker name and password from *mattock*. Finally fill in the hostinfo files as described [here](https://docs.buildbot.net/current/manual/installation/worker.html).

Now you can launch the buildbot worker:

Check */root/buildbot/twistd.log* to see if the worker was able to successfully connect to the master. Typically the error you see is an authentication issue:

This can mean three things:

* Your worker name is wrong

* Your worker password is wrong

* You worker has not yet been added to the buildmaster

If advanced topics and troubleshooting help please refer to the official Buildbot documentation:

* https://docs.buildbot.net/current/manual/installation/worker.html

* https://docs.buildbot.net/current/manual/installation/installation.html

**NOTE:** these instructions have not been tested with the current (3.x) version of Buildbot, but probably still work.

Running buildbot as a root user is easiest, but you can tighten down the setup by giving a normal user limited super-user privileges.

* Create the user, e.g. *buildbot*

* Create the buildslave instance ("Setting up a new buildslave instance") as that user, to a user-editable location (e.g. */home/buildbot/openvpn*)

* Uncomment the *RUN_SUDO=sudo* line in *t_client.rc*

* Create a sudoers snippet in */etc/sudoers.d* with the command *visudo -f /etc/sudoers.d/buildbot*

The buildbot user needs to run two commands as *root*: 'kill' and the 'openvpn' executable inside the various directories of the buildslave project. An example visudo line would be:

OpenVPN project's workers run connectivity tests against several OpenVPN test servers on every commit. Due to these tests the openvpn instances launched by buildbot need to run as *root*, or you need to configure *sudo* properly as described below.

As the tests connect to remote OpenVPN servers you will need test certificates and a *t_client.rc* config file from the buildmaster admins (see above). Once you've have the files, put them to */home/buildbot*:

Note that these files are separate from the community VPN certificates. Make sure that the above files

* are named exactly as shown above or buildbot won't find them and will fail

* are readable by root (or the non-privileged buildbot)

* have strict enough permissions to keep OpenVPN happy

* 600 for *test-client.key*

* 644 for other files

After the keys are installed, a few more steps are required:

* Install *fping* and *fping6*, which the tests use to test for basic connectivity

* Allow outbound traffic through the local firewall to

* OpenVPN Git repository (git://git.code.sf.net)

* Cmocka Git repository (git://git.cryptomilk.org)

* t_client test servers (listening on UDP and TCP ports 51194-51199)

Once you're finished doing all of this, contact the buildmaster admins so that they can force a build (and the associated connectivity tests) on your worker. The first build is expected to fail, because the t_client.rc you were given first is a generic one. After the first build you can fix the values in *t_client.rc* file by looking at client test logs in *<buildslave-dir/build/<buildername>/tests/t_client_<buildername>-<id>*. For example, for the Ubuntu 12.04 worker the build logs were in this directory:

* /var/lib/buildbot/slaves/openvpn/build-ubuntu-1204-i386-stable-master/build/tests/t_client-ubuntu-1204-i386-20141217-160636

The files *<n>:ifconfig_route.txt* contain ifconfig output after OpenVPN had launched in test number <n>. Check what IPv4 and IPv6 addresses the server gave back, and edit your */home/buildbot/t_client.rc* to match. Once you've fixed all the tests ask a buildmaster maintainer to trigger a new build and see if all works as expected. Rinse and repeat as necessary.

You should run as many tests as possible. If your ISP supports IPv6, a reasonable set of tests is this:

If you lack IPV6 transport support, then use

instead.

When connected to the VPN Buildmaster is available at http://buildbot-host.openvpn.in:8010

Buildmaster is configured to use a local Git repository on its Docker host. To push to it use an SSH URL:

To pull you can use the Git protocol:

In case your build fails, try running the same build steps manually to see what the problem is.

The workers consume large amount of diskspace and inodes:

This happens because workers leaves obsolete build directories laying around, removing them can help free up some space and inodes:

On Ubuntu it is fairly important to remove useless kernels and kernel headers periodically:

The worker directory (e.g. */var/lib/buildbot/slaves/openvpn*) contains a logfile, *twistd.log*, which will help you figure out what went wrong. Usually there is an authentication problem, so double-check the worker username and password, and edit *buildbot.tac* as necessary.