Table of Contents

1. Introduction
2. OpenVPN projects being built by Buildbot
3. Creating a VM for a worker
4. Ensure that manual building works correctly
5. Setting up the VPN connection
   1. Testing OpenVPN connectivity
6. Setting up a worker
   1. Configuring the buildslave for connectivity tests
7. List of existing buildslaves
8. List of build permutations
9. Troubleshooting
   1. Source checkout
   2. Additional Enviroment variables
   3. Buildslaves running out of disk or inodes
   4. Buildslave not connecting to buildmaster

Introduction

OpenVPN project uses ​Buildbot to help increase code quality. Buildbot is a Python application that can work in either master or worker mode. The buildmaster is the core server which accepts connections from workers and tells them what they should do. Typically the clients fetch latest sources and reports any build problems to buildbot which in turn informs developers via email. In software engineering this is called ​Continous integration and helps prevent build problems go unnoticed for extended time periods. The clients (workers) can and should run on a variety of hardware / OS platforms. For the server (buildmaster) the OS choice is largely irrelevant. Buildbot is described in more detail in the ​Buildbot manual.

As the number of workers can easily get out of hand, the OpenVPN project can make use of your help. If you're interested in donating a buildslave please contact the buildmaster admins:

• Gert (cron2 on IRC)
• Samuli (mattock on IRC)

NOTE: The workers need root access to be able to connect to the t_client.sh test servers using OpenVPN. Moreover, anyone who controls the buildmaster can do whatever he wants on the workers by simply modifying the build procedure. For this reason you should only run workers on expendable virtual machines or containers. That said, you can grant limited sudo privileges to a normal user to run buildbot as a normal user: see section "Running builtdot as non-root user" for details.

OpenVPN projects being built by Buildbot

Currently buildbot builds the following codebases:

• openvpn
• openvpn3
• openvpn3-linux
• ovpn-dco

You can decide which ones you want your buildbot worker to build.

Creating a VM for a worker

A Linux VM will probably need about 1024MB of memory.The VM should have at least 8GB of diskspace, but adding a bit more helps avoid problems later on.

Ensure that manual building works correctly

The first step is to ensure you can build all the codebases you need. If manually building does not work, it cannot work in buildbot, either. Please follow the instructions for each codebase on how to build.

Setting up the VPN connection

Our buildmaster is accessible only via the community VPN. Please ask mattock on IRC for VPN config for your worker. You will get the config in Signal or in an GnuPG-encrypted email. The community VPN server pushes DNS servers as well, so you may want to process those if you want to use DNS names instead of IP addresses.

Testing OpenVPN connectivity

A simple ping test should be enough to verify that your worker can reach buildmaster:

$ ping buildbot-host.openvpn.in
$ ping 10.18.0.59

If both work, it means your VPN client is properly configured. If pinging by name fails, but by IP works, then your VPN client is not processing DNS settings pushed by the OpenVPN server properly. But you can still point your worker to the IP, so it is not a blocker.

Setting up a worker

Running buildbot as non-root user

Running buildbot as a root user is easiest, but you can tighten down the setup by giving a normal user limited super-user privileges.

• Create the user, e.g. buildbot
• Create the buildslave instance ("Setting up a new buildslave instance") as that user, to a user-editable location (e.g. /home/buildbot/openvpn)
• Uncomment the RUN_SUDO=sudo line in t_client.rc
• Create a sudoers snippet in /etc/sudoers.d with the command visudo -f /etc/sudoers.d/buildbot

The buildbot user needs to run two commands as root: 'kill' and the 'openvpn' executable inside the various directories of the buildslave project. An example visudo line would be:

buildbot ALL=(root) NOPASSWD: /usr/bin/kill,/home/buildbot/<build_slave_dir>/*/build/src/openvpn/openvpn,/home/buildbot/<build_slave_dir>/*/build/tests/unit_tests/openvpn/networking_testdriver

Configuring the buildslave for connectivity tests

OpenVPN project's buildslaves run connectivity tests against several OpenVPN test servers on every commit. Due to these tests the openvpn instances launched by buildbot need to run as root, or you need to configure sudo properly as described below.

As the tests connect to remote OpenVPN servers you will need test certificates and a t_client.rc config file from the buildmaster admins (see above). Once you've have the files, put them to /home/buildbot:

$ tree /home/buildbot
/home/buildbot
├── t_client.rc
├── test-ca.crt
├── test-client.crt
├── test-client.key
└── test-ta.key

0 directories, 5 files

Note that these files are separate from the community VPN certificates. Make sure that the above files

• are named exactly as shown above or buildbot won't find them and will fail
• are readable by root (or the non-privileged buildbot)
• have strict enough permissions to keep OpenVPN happy
  • 600 for test-client.key
  • 644 for other files

After the keys are installed, a few more steps are required:

• Install fping and fping6, which the tests use to test for basic connectivity
• Allow outbound traffic through the local firewall to
  • OpenVPN Git repository (​git://git.code.sf.net)
  • Cmocka Git repository (​git://git.cryptomilk.org)
  • t_client test servers (listening on UDP and TCP ports 51194-51199)

Once you're finished doing all of this, contact the buildmaster admins so that they can force a build (and the associated connectivity tests) on your buildslave. The first build is expected to fail, because the t_client.rc you were given first is a generic one. After the first build you can fix the values in t_client.rc file by looking at client test logs in <buildslave-dir/build/<buildername>/tests/t_client_<buildername>-<id>. For example, for the Ubuntu 12.04 buildslave the build logs were in this directory:

• /var/lib/buildbot/slaves/openvpn/build-ubuntu-1204-i386-stable-master/build/tests/t_client-ubuntu-1204-i386-20141217-160636

The files <n>:ifconfig_route.txt contain ifconfig output after OpenVPN had launched in test number <n>. Check what IPv4 and IPv6 addresses the server gave back, and edit your /home/buildbot/t_client.rc to match. Once you've fixed all the tests ask a buildmaster maintainer to trigger a new build and see if all works as expected. Rinse and repeat as necessary.

You should run as many tests as possible. If your ISP supports IPv6, a reasonable set of tests is this:

TEST_RUN_LIST="1 1a 2 2a 2b 2c 3 4 4a 5 6"

If you lack IPV6 transport support, then use

TEST_RUN_LIST="1 2 3 4 5 6"

instead.

List of existing buildslaves

Here's a comprehensive list of Buildslaves already running (as of Apr 2014). There can be several buildslaves that have the same OS/architecture combination, but this is seldom necessary.

Operating system	Version	Architecture	24/7	Connection tests	Provided by	Notes
Debian	7 (wheezy)	i386	Yes	Yes1	mattock
Ubuntu	12.04	i386	Yes	Yes1	mattock
Ubuntu	14.04	amd64	Yes	Yes1	mattock
Ubuntu	16.04	amd64	Yes	Yes1	mattock
CentOS	6	amd64	No	Yes1	mattock
Fedora	24	amd64	No	Yes1	mattock
FreeBSD	7.4	amd64	Yes	Yes	cron2
FreeBSD	9.3	amd64	Yes	Yes	cron2
FreeBSD	10.3	amd64	Yes	Yes	cron2
OpenBSD	4.9	i386	Yes	Yes	cron2
OpenBSD	6.0	i386	Yes	Yes	cron2
NetBSD	5.1	amd64	Yes	Yes	cron2
NetBSD	7.0.1	amd64	Yes	Yes	cron2
Opensolaris	10	i386	Yes	Yes	cron2
MacOS X	amd64		No	Yes	plaisthos
Archlinux		amd64	No2	Yes1	tincantech
CentOS	7	amd64	No2	Yes1	tincantech

Notes:

1. These buildslaves run tests 1-6 without the "a" variants. In practice this means that proxy and IPv6 transport are untested.

2. These buildslaves are Virtual Machines and may be temporarily unavailable.

List of build permutations

Each commit triggers several builds of OpenVPN, each with a different set of configure options:

config_opt_combos = [
    "",
    "--disable-crypto",
    "--disable-crypto --disable-lzo",
    "--disable-crypto --disable-lzo --disable-management",
    "--disable-crypto --disable-management",
    "--disable-lz4",
    "--disable-lzo",
    "--disable-lzo --disable-lz4 --enable-comp-stub",
    "--disable-lzo --disable-management",
    "--disable-management",
    "--disable-server --enable-small",
    "--enable-small",
    "--with-crypto-library=mbedtls --enable-crypto",
]

Most of the builds just build OpenVPN to ensure the buildsystem works. However, two of them also trigger a suite of t_client.sh connectivity tests:

• ./configure
• ./configure --with-crypto-library=mbedtls --enable-crypto

The connectivity tests help ensure that basic functionality in OpenVPN is never completely broken.

Troubleshooting

In case your build fails, try running the same build steps manually to see what the problem is.

Source checkout

If you're running into odd problems with buildslave Git checkouts you can try manually running the same commands the buildslave would run. The exact steps can be found from the buildslave's Git logs (in the buildmaster web interface), but they should be similar to these:

$ mkdir /tmp/openvpntest
$ cd /tmp/openvpntest
$ git init
$ git fetch -t git://openvpn.git.sourceforge.net/gitroot/openvpn/openvpn.git +master
$ git reset --hard FETCH_HEAD
$ git branch -M master

If you're unable to run these commands manually then the buildslave won't be able to do it either.

Additional Enviroment variables

It is possible to add additional environment variables to buildbot.tac:

for e in  ['LZO_CFLAGS=-I/usr/local/include', 
           'LZO_LIBS=-llzo2 -L/usr/local/lib',
	   'MBEDTLS_CFLAGS=-I/usr/local/Cellar/mbedtls/2.3.0_1/include',
	   'MBEDTLS_LIBS=-L/usr/local/Cellar/mbedtls/2.3.0_1/lib/ -lmbedtls -lmbedx509 -lmbedcrypto',
	   'OPENSSL_CFLAGS=-I/usr/local/Cellar/openssl/1.0.2h_1/include/',
	   'OPENSSL_LIBS=-L/usr/local/Cellar/openssl/1.0.2h_1/lib/ -lcrypto -lssl']:
	   var, value = e.split('=',1)
	   os.environ[var]=value

This method is a bit hackish but works fine.

Buildslaves running out of disk or inodes

The buildslaves consume large amount of diskspace and inodes:

$ df -m|grep -E '(^Filesystem|vda1)'
Filesystem     1M-blocks  Used Available Use% Mounted on
/dev/vda1           9196  4342      4365  50% /

$ df -i|grep -E '(^Filesystem|vda1)'
Filesystem     Inodes  IUsed  IFree IUse% Mounted on
/dev/vda1      606208 274616 331592   46% /

Because buildslave leaves obsolete build directories laying around, removing them can help free up some space and inodes:

$ rm -rf /var/lib/buildbot/slaves/openvpn/build-*

On Ubuntu it is fairly important to remove useless kernels and kernel headers periodically:

$ apt-get autoremove

Buildslave not connecting to buildmaster

The buildslave directory (e.g. /var/lib/buildbot/slaves/openvpn) contains a logfile, twistd.log, which will help you figure out what went wrong. Usually there is an authentication problem, so double-check the buildslave username and password, and edit buildbot.tac as necessary.