Table of Contents
Introduction
OpenVPN project uses Buildbot to help increase code quality. Buildbot is a Python application that can work in either master or worker mode. The buildmaster is the core server which accepts connections from workers and tells them what they should do. Typically the clients fetch latest sources and reports any build problems to buildbot which in turn informs developers via email. In software engineering this is called Continous integration and helps prevent build problems go unnoticed for extended time periods. The clients (workers) can and should run on a variety of hardware / OS platforms. For the server (buildmaster) the OS choice is largely irrelevant. Buildbot is described in more detail in the Buildbot manual.
As the number of workers can easily get out of hand, the OpenVPN project can make use of your help. If you're interested in donating a buildslave please contact the buildmaster admins:
- Gert (cron2 on IRC)
- Samuli (mattock on IRC)
NOTE: The workers need root access to be able to connect to the t_client.sh test servers using OpenVPN. Moreover, anyone who controls the buildmaster can do whatever he wants on the workers by simply modifying the build procedure. For this reason you should only run workers on expendable virtual machines or containers. That said, you can grant limited sudo privileges to a normal user to run buildbot as a normal user: see section "Running builtdot as non-root user" for details.
OpenVPN projects being built by Buildbot
Currently buildbot builds the following projects:
- openvpn
- openvpn3
- openvpn3-linux
- ovpn-dco
You can decide which ones you want your buildbot worker to build.
List of existing workers
Here's a comprehensive list of workers already running (as of June 2022):
- debian-10
- debian-11
- debian-unstable
- arch
- fedora-34
- opensuse-leap-15
- ubuntu-1804
- ubuntu-2004
- ubuntu-2110
- cron2-fbsd74
- cron2-fbsd11
- cron2-fbsd12
- cron2-fbsd13
- cron2-nbsd81
- cron2-obsd68
- cron2-oi2019
Creating a VM for a worker
A Linux VM will probably need about 1024MB of memory.The VM should have at least 8GB of diskspace, but adding a bit more helps avoid problems later on.
Ensure that manual building works correctly
The first step is to ensure you can build all the projects you wish to build. If manually building does not work, it cannot work in buildbot, either. Please follow the instructions for each codebase on how to build.
Setting up the VPN connection
Our buildmaster is accessible only via the community VPN. Please ask mattock on IRC for VPN config for your worker. You will get the config in Signal or in an GnuPG-encrypted email. The community VPN server pushes DNS servers as well, so you may want to process those if you want to use DNS names instead of IP addresses.
Testing OpenVPN connectivity
A simple ping test should be enough to verify that your worker can reach buildmaster:
$ ping buildbot-host.openvpn.in $ ping 10.18.0.59
If both work, it means your VPN client is properly configured. If pinging by name fails, but by IP works, then your VPN client is not processing DNS settings pushed by the OpenVPN server properly. But you can still point your worker to the IP, so it is not a blocker.
Setting up a worker
Simplistic setup
Note that Buildbot no longer supports Python 2.x, so ensure that your to-be worker has a working Python 3 installation.
A simplistic procedure for Ubuntu 20.04 is described below. As t_client tests need root-level privileges anyways, here we show how to run a worker as root:
$ sudo -i $ apt-get install python3-pip $ pip3 install buildbot-worker $ mkdir /root/buildbot $ buildbot-worker create-worker /root/buildbot buildbot-host.openvpn.in:9989 worker-name worker-password
You can get the worker name and password from mattock. Finally fill in the hostinfo files as described here.
Now you can launch the buildbot worker:
$ buildbot-worker start /root/buildbot
Check /root/buildbot/twistd.log to see if the worker was able to successfully connect to the master. Typically the error you see is an authentication issue:
2022-06-29 11:46:44+0000 [Broker,client] unauthorized login; check worker name and password
This can mean three things:
- Your worker name is wrong
- Your worker password is wrong
- You worker has not yet been added to the buildmaster
If advanced topics and troubleshooting help please refer to the official Buildbot documentation:
- https://docs.buildbot.net/current/manual/installation/worker.html
- https://docs.buildbot.net/current/manual/installation/installation.html
Running buildbot as non-root user
NOTE: these instructions have not been tested with the current (3.x) version of Buildbot, but probably still work.
Running buildbot as a root user is easiest, but you can tighten down the setup by giving a normal user limited super-user privileges.
- Create the user, e.g. buildbot
- Create the buildslave instance ("Setting up a new buildslave instance") as that user, to a user-editable location (e.g. /home/buildbot/openvpn)
- Uncomment the RUN_SUDO=sudo line in t_client.rc
- Create a sudoers snippet in /etc/sudoers.d with the command visudo -f /etc/sudoers.d/buildbot
The buildbot user needs to run two commands as root: 'kill' and the 'openvpn' executable inside the various directories of the buildslave project. An example visudo line would be:
buildbot ALL=(root) NOPASSWD: /usr/bin/kill,/home/buildbot/<build_slave_dir>/*/build/src/openvpn/openvpn,/home/buildbot/<build_slave_dir>/*/build/tests/unit_tests/openvpn/networking_testdriver
Configuring a worker for connectivity tests
OpenVPN project's workers run connectivity tests against several OpenVPN test servers on every commit. Due to these tests the openvpn instances launched by buildbot need to run as root, or you need to configure sudo properly as described below.
As the tests connect to remote OpenVPN servers you will need test certificates and a t_client.rc config file from the buildmaster admins (see above). Once you've have the files, put them to /home/buildbot:
$ tree /home/buildbot /home/buildbot ├── t_client.rc ├── test-ca.crt ├── test-client.crt ├── test-client.key └── test-ta.key 0 directories, 5 files
Note that these files are separate from the community VPN certificates. Make sure that the above files
- are named exactly as shown above or buildbot won't find them and will fail
- are readable by root (or the non-privileged buildbot)
- have strict enough permissions to keep OpenVPN happy
- 600 for test-client.key
- 644 for other files
After the keys are installed, a few more steps are required:
- Install fping and fping6, which the tests use to test for basic connectivity
- Allow outbound traffic through the local firewall to
- OpenVPN Git repository (git://git.code.sf.net)
- Cmocka Git repository (git://git.cryptomilk.org)
- t_client test servers (listening on UDP and TCP ports 51194-51199)
Once you're finished doing all of this, contact the buildmaster admins so that they can force a build (and the associated connectivity tests) on your worker. The first build is expected to fail, because the t_client.rc you were given first is a generic one. After the first build you can fix the values in t_client.rc file by looking at client test logs in <buildslave-dir/build/<buildername>/tests/t_client_<buildername>-<id>. For example, for the Ubuntu 12.04 worker the build logs were in this directory:
- /var/lib/buildbot/slaves/openvpn/build-ubuntu-1204-i386-stable-master/build/tests/t_client-ubuntu-1204-i386-20141217-160636
The files <n>:ifconfig_route.txt contain ifconfig output after OpenVPN had launched in test number <n>. Check what IPv4 and IPv6 addresses the server gave back, and edit your /home/buildbot/t_client.rc to match. Once you've fixed all the tests ask a buildmaster maintainer to trigger a new build and see if all works as expected. Rinse and repeat as necessary.
You should run as many tests as possible. If your ISP supports IPv6, a reasonable set of tests is this:
TEST_RUN_LIST="1 1a 2 2a 2b 2c 3 4 4a 5 6"
If you lack IPV6 transport support, then use
TEST_RUN_LIST="1 2 3 4 5 6"
instead.
Accessing buildmaster webui
When connected to the VPN Buildmaster is available at http://buildbot-host.openvpn.in:8010
Using the local Git repo on buildmaster
Buildmaster is configured to use a local Git repository on its Docker host. To push to it use an SSH URL:
git+ssh://myuser@10.18.0.59/var/lib/repos/openvpn
To pull you can use the Git protocol:
git://10.18.0.59:9418/openvpn
Troubleshooting
Build failures
In case your build fails, try running the same build steps manually to see what the problem is.
Workers running out of disk or inodes
The workers consume large amount of diskspace and inodes:
$ df -m|grep -E '(^Filesystem|vda1)' Filesystem 1M-blocks Used Available Use% Mounted on /dev/vda1 9196 4342 4365 50% / $ df -i|grep -E '(^Filesystem|vda1)' Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vda1 606208 274616 331592 46% /
This happens because workers leaves obsolete build directories laying around, removing them can help free up some space and inodes:
$ rm -rf /var/lib/buildbot/slaves/openvpn/build-*
On Ubuntu it is fairly important to remove useless kernels and kernel headers periodically:
$ apt-get autoremove
Worker not connecting to buildmaster
The worker directory (e.g. /var/lib/buildbot/slaves/openvpn) contains a logfile, twistd.log, which will help you figure out what went wrong. Usually there is an authentication problem, so double-check the worker username and password, and edit buildbot.tac as necessary.
