Version 53 (modified by Samuli Seppänen, 4 years ago) (diff)

Update list of buildslaves


OpenVPN project uses Buildbot (current version 0.8.5) to help increase code quality. Buildbot is a Python application that can work in either master or slave mode. The buildmaster is the core server which accepts connections from buildslaves and tells them what they should do. Typically the clients fetch latest sources and reports any build problems to buildbot which in turn informs developers via email. In software engineering this is called Continous integration and helps prevent build problems go unnoticed for extended time periods. The clients (buildslaves) can and should run on a variety of hardware / OS platforms. For the server (buildmaster) the OS choice is largely irrelevant. Buildbot is described in more detail in the Buildbot manual.

As the number of buildslaves can easily get out of hand, the OpenVPN project can make use of your help. If you're interested in donating a buildslave please contact the buildmaster admins:

  • Gert (cron2 on IRC)
  • Samuli (mattock on IRC)

Setting up the VPN connection

Our buildbot (buildmaster) is accessible only via OpenVPN. To connect your buildslave to it you need:

  1. Active OpenVPN community services user account
  2. A properly configured OpenVPN client
  3. Certificate authority (CA) key for the VPN
  4. TLS-auth (TA) key for the VPN
  5. Membership in the appropriate LDAP group

You can do 1. yourself, but for the rest you need to contact buildmaster admins (see above).

Setting up a buildslave

Installing buildbot

Installing buildbot from distro repositories

Many Linux distributions have buildbot in their package repositories. Using the distro version is probably the easiest approach. Buildbot versions 0.7.x and 0.8.x are known to work with our current (0.8.5) buildmaster.

Installing buildbot using easy_install

A fairly easy and somewhat OS-agnostic way to install buildbot is to use easy_install: it should be available your OS'es software repository. Using easy_install will keep buildslave isolated from your OS'es package management and take care of dependencies for you. It is easiest (but not necessary) to run easy_install as root:

First switch to root account, e.g. using sudo or su:

$ sudo -s

Next test that easy_install should work as expected:

$ easy_install -n buildbot==0.8.5

If all went well, install buildbot for real:

$ easy_install buildbot==0.8.5

Once buildslave is installed you can and should switch to an ordinary user account.

Configuring the buildslave

Buildslave configuration is covered thoroughly in the the buildbot manual. Basically you just step through the "Creating a buildslave" section in the buildbot manual except that you could skip step 1 ("Set up the account") because the connectivity tests require root access.

Step-by-step, the following is needed:

  1. create a user ("buildbot")
  2. create a buildslave configuration, in a subdirectory of buildbot's home directory
    user$ sudo su - buildbot
    buildbot$ buildslave create-slave ~/build-openvpn <buildbot name> <buildbot pass>

this will create a subdirectory "build-openvpn/" which has a *.tac file with the buildbot config (so, username, buildmaster, etc. can be edited later on). <buildbot-name> and <buildbot-pass> need to be coordinated with the buildmaster admins (mattock).

  1. ensure the community VPN is up (ping works)
  2. start the buildslave
    buildbot$ buildslave start ~/build-openvpn
  3. ensure that community VPN and buildslave get started at boot time...

In case you need to run the buildslave as a limited user you may follow these steps:

  • create the user, e.g. 'buildbot'.
  • uncomment the RUN_SUDO=sudo line in t_client.rc.
  • create a sudoers snippet in /etc/sudoers.d with the command 'visudo -f /etc/sudoers.d/buildbot'.

The buildbot user needs to run two commands as root: 'kill' and the 'openvpn' executable inside the various directories of the buildslave project.

An example visudo line would be:

buildbot ALL=(root) NOPASSWD: /usr/bin/kill,/home/buildbot/<build_slave_dir>/*/build/src/openvpn/openvpn

Installing OpenSSL/PolarSSL development files

We test building OpenVPN with both OpenSSL and PolarSSL backends, so you need to both of them installed on your buildslave:

  • For OpenSSL you can typically use whatever development package (e.g. libssl-dev) you operating system provides, because it's APIs are fairly stable.
  • With PolarSSL the best bet is to get latest stable version from the PolarSSL homepage, because the APIs change quite often and OpenVPN tends to track the latest one.

Configuring the buildslave for connectivity tests

OpenVPN project's buildslaves run connectivity tests against several OpenVPN test servers once per on every commit. Due to these tests the openvpn instances launched by buildbot need to run as root.

As the tests connect to remote OpenVPN servers you will need test certificates and a t_client.rc config file from the buildmaster admins (see above). Once you've have the files, put them to /home/buildbot:

$ tree /home/buildbot
├── t_client.rc
├── test-ca.crt
├── test-client.crt
├── test-client.key
└── test-ta.key

0 directories, 5 files

Now make sure that the files

  • are named exactly as shown above or buildbot won't find them and will fail
  • are readable by root
  • have strict enough permissions to keep OpenVPN happy
    • 600 for test-client.key
    • 644 for other files

Finally install fping and fping6, which the tests use to test for basic connectivity.

Once you're finished doing all of this, contact the buildmaster admins so that they can force a build (and the associated connectivity tests) on your buildslave. The first build is expected to fail, because the t_client.rc you were given first is a generic one. After the first build you can fix the values in t_client.rc file by looking at client test logs in <buildslave-dir/build/<buildername>/tests/t_client_<buildername>-<id>. For example, for the Ubuntu 12.04 buildslave the build logs were in this directory:

  • /var/lib/buildbot/slaves/openvpn/build-ubuntu-1204-i386-stable-master/build/tests/t_client-ubuntu-1204-i386-20141217-160636

Look into files 1:ifconfig_route.txt and 2:ifconfig_route.txt to see which IPv4 and IPv6 addresses the test VPN servers gave to the OpenVPN client and adjust t_client.rc to match those values. After that ask somebody to trigger a new build and see if all works as expected. Rinse and repeat as necessary.

List of existing buildslaves

Here's a comprehensive list of Buildslaves already running (as of Apr 2014). There can be several buildslaves that have the same OS/architecture combination, but this is seldom necessary.

Operating systemVersionArchitecture24/7Connection testsProvided byNotes
Debian7 (wheezy)i386YesYes1mattock
MacOS Xamd64NoYesplaisthos


  1. These buildslaves run tests 1-6 without the "a" variants. In practice this means that proxy and IPv6 transport are untested.

List of build permutations

There are too many to list. However, the builds that use default build flags also trigger connectivity tests.


In case your build fails, try running the same build steps manually to see what the problem is.

Source checkout

If you're running into odd problems with buildslave Git checkouts you can try manually running the same commands the buildslave would run. The exact steps can be found from the buildslave's Git logs (in the buildmaster web interface), but they should be similar to these:

$ mkdir /tmp/openvpntest
$ cd /tmp/openvpntest
$ git init
$ git fetch -t git:// +master
$ git reset --hard FETCH_HEAD
$ git branch -M master

If you're unable to run these commands manually then the buildslave won't be able to do it either.

Additional Enviroment variables

Add this to buildbot.tac, a bit hackish but works fine:

for e in  ['LZO_CFLAGS=-I/usr/local/include', 
           'LZO_LIBS=-llzo2 -L/usr/local/lib',
	   'MBEDTLS_LIBS=-L/usr/local/Cellar/mbedtls/2.3.0_1/lib/ -lmbedtls -lmbedx509 -lmbedcrypto',
	   'OPENSSL_LIBS=-L/usr/local/Cellar/openssl/1.0.2h_1/lib/ -lcrypto -lssl']:
	   var, value = e.split('=',1)