Table of Contents
- Creating a VM for the buildslave
- Ensure that OpenVPN builds correctly
- Setting up the VPN connection
- Setting up a buildslave
- List of existing buildslaves
- List of build permutations
OpenVPN project uses Buildbot (current version 0.8.5) to help increase code quality. Buildbot is a Python application that can work in either master or slave mode. The buildmaster is the core server which accepts connections from buildslaves and tells them what they should do. Typically the clients fetch latest sources and reports any build problems to buildbot which in turn informs developers via email. In software engineering this is called Continous integration and helps prevent build problems go unnoticed for extended time periods. The clients (buildslaves) can and should run on a variety of hardware / OS platforms. For the server (buildmaster) the OS choice is largely irrelevant. Buildbot is described in more detail in the Buildbot manual.
As the number of buildslaves can easily get out of hand, the OpenVPN project can make use of your help. If you're interested in donating a buildslave please contact the buildmaster admins:
- Gert (cron2 on IRC)
- Samuli (mattock on IRC)
NOTE: The buildslave needs root access to be able to connect to the t_client.sh test servers using OpenVPN. Moreover, anyone who controls the buildmaster can do whatever he wants on the buildslaves by simply modifying the build procedure. For this reason you should only run buildslaves on expendable virtual machines. That said, you can grant limited sudo privileges to a normal user to run buildbot as a normal user: see section "Running builtdot as non-root user" for details.
Creating a VM for the buildslave
A Linux VM will probably need at least 768MB of memory. Some distros such as Fedora 24 suggests 1024MB as the bare minimum. The VM should have at least 8GB of diskspace, but adding a bit more helps avoid problems later on.
Ensure that OpenVPN builds correctly
The first step is to ensure you can build OpenVPN on your buildslave VM. On Debian/Ubuntu this takes care of most of the dependencies (from here):
$ apt-get update $ apt-get install liblzo2-dev libssl-dev libpam-dev libpkcs11-helper-dev libtool autoconf make cmake
On Fedora you'd do
$ dnf install lzo-devel openssl-devel pam-devel pkcs11-helper-devel gnutls-devel autoconf libtool make cmake
And on RedHat/CentOS/Scientific Linux you'd do
$ yum install lzo-devel openssl-devel pam-devel pkcs11-helper-devel gnutls-devel autoconf libtool make cmake
On *BSD derivatives the procedure is roughly similar.
On top of this you need to build and install mbedTLS. As OpenVPN tends to follow mbedTLS's rapidly changing APIs, installing the latest version is usually the best choice. Follow the instructions in the release package on how to build mbedTLS on your system.
Once you've installed all the build dependencies try building OpenVPN. You should test at least
$ ./configure --with-crypto-library=mbedtls --enable-crypto
The former tests OpenSSL builds and the latter tests mbedTLS builds. If these two builds succeed, there is high probability that your buildslave will not fail miserably when put to work.
Setting up the VPN connection
Our buildmaster is accessible only via OpenVPN. The following sections describe what you need to to connect to the VPN.
Creating the community user account
First you need an OpenVPN community services user account. Please use a dedicated community user account with a strong password for your buildslave. For clarity, you should use the following naming convention:
There is no need to create a dedicated email account for the buildslave: using your personal email address is just fine. You can use your own firstname and lastname for the account, or just fake them.
Getting membership in the appropriate community LDAP group
Once you have created the community account for your buildslave, let the buildmaster maintainers know what it is called, so that they can add it to the correct LDAP group. Without proper group membership your buildslave's OpenVPN instance will not be able to connect to the community VPN server.
Getting the OpenVPN configuration file
Buildmaster maintainers will provide you with an OpenVPN configuration with inlined ca.crt and ta.key. You can use pretty much any semi-recent OpenVPN version to connect to the community VPN server. Place the configuration file to the usual place, e.g. /etc/openvpn/buildslave.conf.
Non-systemd distros typically blindly launch connections for every .conf file they find under /etc/openvpn at boot-time. Launching the VPN connection manually is as simple as
$ service openvpn start
Systemd distros will not automatically start any single VPN connection without a bit of trickery:
$ ln -s /firstname.lastname@example.org /usr/lib/systemd/system/openvpn@.service
Launching a VPN connection on systemd distros is fairly straightforward:
$ systemctl start openvpn@buildslave
Note that the above will only work if the OpenVPN configuration file is in /etc/openvpn/buildslave.conf.
Testing OpenVPN connectivity
A simple ping test should be enough to verify that your buildslave can reach buildmaster:
$ ping 10.177.36.101
Setting up a buildslave
Buildbot versions 0.7.x and 0.8.x are known to work with our current (0.8.5) buildmaster. Buildbot 0.9.x is known _not_ to work. Generic setup instructions are available in the the buildbot manual.
Common setup tasks
These tasks are out of order and referred to in the setup instructions. Please do not run these right now.
Setting up a new buildslave instance
This command is the same regardless of how you install or setup the buildslave, so it is shown here. You need to run this command after you have installed buildbot-slave, or it will fail.
$ buildslave create-slave /var/lib/buildbot/slaves/openvpn 10.177.36.101:9989 <slavename>
The slavename translates to slave-<youralias>-<operatingsystem>-<version>-<architecture> <buildslave-password>.
This will create a buildslave instance to /var/lib/buildbot/slaves/openvpn. You do not need to use that directory, but Debian and Ubuntu use it by default, and it seems like a reasonable place for RedHat derivatives as well.
The buildslave directory has a *.tac file with the buildbot config, so the settings can be edited later on. You can get the <buildslave-password> from the buildmaster admins.
Starting the buildslave at boot-time
Debian and Ubuntu bundle a buildslave init script with their buildbot-slave package. On other distributions (or when you installed using easy_install) you need to take care of starting buildslave on boot yourself. If you happen to be using monit then you can use this monit fragment:
check process buildslave-openvpn with pidfile /var/lib/buildbot/slaves/openvpn/twistd.pid start "/usr/bin/buildslave start /var/lib/buildbot/slaves/openvpn" stop "/usr/bin/buildslave stop /var/lib/buildbot/slaves/openvpn"
Monit not only starts the buildslave, but also ensures that it is running all the time.
Another approach is to add
$ buildslave start /var/lib/buildbot/slaves/openvpn
to /etc/rc.local or similar startup script. Yet another option is to make cron periodically test if buildslave is running, and to launch it if it's not.
Running buildbot as non-root user
Running buildbot as a root user is easiest, but you can tighten down the setup by giving a normal user limited super-user privileges.
- Create the user, e.g. buildbot
- Create the buildslave instance ("Setting up a new buildslave instance") as that user, to a user-editable location (e.g. /home/buildbot/openvpn)
- Uncomment the RUN_SUDO=sudo line in t_client.rc
- Create a sudoers snippet in /etc/sudoers.d with the command visudo -f /etc/sudoers.d/buildbot
The buildbot user needs to run two commands as root: 'kill' and the 'openvpn' executable inside the various directories of the buildslave project. An example visudo line would be:
buildbot ALL=(root) NOPASSWD: /usr/bin/kill,/home/buildbot/<build_slave_dir>/*/build/src/openvpn/openvpn
Buildslave setup using distro packages on Debian/Ubuntu
Debian and Ubuntu have buildbot-slave in their repositories. First become a root user using sudo or su. Then install the buildslave application with
$ apt-get update && apt-get install buildbot-slave
After the install you have to create the buildslave instance as described above in "Setting up a new buildslave instance" section.
The newly created buildbot instance has to be enabled by editing /etc/default/buildslave:
SLAVE_RUNNER=/usr/bin/buildslave # NOTE: SLAVE_ENABLED has changed its behaviour in version 0.8.4. Use # 'true|yes|1' to enable instance and 'false|no|0' to disable. Other # values will be considered as syntax error. SLAVE_ENABLED=1 SLAVE_NAME=openvpn # root user is needed, see above for details SLAVE_USER=root SLAVE_BASEDIR=/var/lib/buildbot/slaves/openvpn SLAVE_OPTIONS= SLAVE_PREFIXCMD=
On non-systemd distros you start the buildslave like this:
$ service buildslave start
On systemctl-enabled distros you'd do this instead:
$ systemctl start buildslave
Buildslave setup using distro packages on Fedora
Fedora has buildbot-slave in its repositories, and the installation is a breeze:
$ dnf install buildbot-slave
After the install you have to create the buildslave instance as described above in "Setting up a new buildslave instance" section. Then start the buildslave with
$ buildslave start /var/lib/buildbot/slaves/openvpn
Unfortunately there is no startup script in the buildbot-slave Fedora package, so you will have to create a startup script yourself. See "Starting the buildslave at boot-time" section (above) for details.
Buildslave setup using easy_install
A fairly easy and somewhat OS-agnostic way to install buildbot is to use easy_install: it should be available your OS'es software repository. By using easy_install the buildslave is isolated from your OS'es package management, but dependencies are taken care for you. It is easiest (but not necessary) to run easy_install as root:
First switch to root account (e.g. sudo or su). Then test that easy_install should work as expected:
$ easy_install -n buildbot==0.8.5
If all went well, install buildbot for real:
$ easy_install buildbot==0.8.5
You can install any version that is known to be compatible with our buildmaster (see above)
Next setup a buildslave instance as described in "Setting up a new buildslave instance", and then start the buildslave:
$ buildslave start /var/lib/buildbot/slaves/openvpn
Finally you need to ensure that buildslave starts at boot-time; see "Starting the buildslave at boot-time" section, above, for details.
Configuring the buildslave for connectivity tests
OpenVPN project's buildslaves run connectivity tests against several OpenVPN test servers on every commit. Due to these tests the openvpn instances launched by buildbot need to run as root, or you need to configure sudo properly as described below.
As the tests connect to remote OpenVPN servers you will need test certificates and a t_client.rc config file from the buildmaster admins (see above). Once you've have the files, put them to /home/buildbot:
$ tree /home/buildbot /home/buildbot ├── t_client.rc ├── test-ca.crt ├── test-client.crt ├── test-client.key └── test-ta.key 0 directories, 5 files
Note that these files are separate from the community VPN certificates. Make sure that the above files
- are named exactly as shown above or buildbot won't find them and will fail
- are readable by root (or the non-privileged buildbot)
- have strict enough permissions to keep OpenVPN happy
- 600 for test-client.key
- 644 for other files
After the keys are installed, a few more steps are required:
- Install fping and fping6, which the tests use to test for basic connectivity
- Allow outbound traffic through the local firewall to
Once you're finished doing all of this, contact the buildmaster admins so that they can force a build (and the associated connectivity tests) on your buildslave. The first build is expected to fail, because the t_client.rc you were given first is a generic one. After the first build you can fix the values in t_client.rc file by looking at client test logs in <buildslave-dir/build/<buildername>/tests/t_client_<buildername>-<id>. For example, for the Ubuntu 12.04 buildslave the build logs were in this directory:
The files <n>:ifconfig_route.txt contain ifconfig output after OpenVPN had launched in test number <n>. Check what IPv4 and IPv6 addresses the server gave back, and edit your /home/buildbot/t_client.rc to match. Once you've fixed all the tests ask a buildmaster maintainer to trigger a new build and see if all works as expected. Rinse and repeat as necessary.
You should run as many tests as possible. If your ISP supports IPv6, a reasonable set of tests is this:
TEST_RUN_LIST="1 1a 2 2a 2b 2c 3 4 4a 5 6"
If you lack IPV6 transport support, then use
TEST_RUN_LIST="1 2 3 4 5 6"
List of existing buildslaves
Here's a comprehensive list of Buildslaves already running (as of Apr 2014). There can be several buildslaves that have the same OS/architecture combination, but this is seldom necessary.
|Operating system||Version||Architecture||24/7||Connection tests||Provided by||Notes|
|Archlinux||amd64||No2||Yes1||tincantech||Currently Fails test 6|
|CentOS||7||amd64||No2||Yes1||tincantech||Currently Fails test 6|
|Gentoo||amd64||No2||Yes1||tincantech||Currently Fails test 6|
- These buildslaves run tests 1-6 without the "a" variants. In practice this means that proxy and IPv6 transport are untested.
- These buildslaves are Virtual Machines and may be temporarily unavailable.
List of build permutations
Each commit triggers several builds of OpenVPN, each with a different set of configure options:
config_opt_combos = [ "", "--disable-crypto", "--disable-crypto --disable-lzo", "--disable-crypto --disable-lzo --disable-management", "--disable-crypto --disable-management", "--disable-lz4", "--disable-lzo", "--disable-lzo --disable-lz4 --enable-comp-stub", "--disable-lzo --disable-management", "--disable-management", "--disable-server --enable-small", "--enable-small", "--with-crypto-library=mbedtls --enable-crypto", ]
Most of the builds just build OpenVPN to ensure the buildsystem works. However, two of them also trigger a suite of t_client.sh connectivity tests:
- ./configure --with-crypto-library=mbedtls --enable-crypto
The connectivity tests help ensure that basic functionality in OpenVPN is never completely broken.
In case your build fails, try running the same build steps manually to see what the problem is.
If you're running into odd problems with buildslave Git checkouts you can try manually running the same commands the buildslave would run. The exact steps can be found from the buildslave's Git logs (in the buildmaster web interface), but they should be similar to these:
$ mkdir /tmp/openvpntest $ cd /tmp/openvpntest $ git init $ git fetch -t git://openvpn.git.sourceforge.net/gitroot/openvpn/openvpn.git +master $ git reset --hard FETCH_HEAD $ git branch -M master
If you're unable to run these commands manually then the buildslave won't be able to do it either.
Additional Enviroment variables
It is possible to add additional environment variables to buildbot.tac:
for e in ['LZO_CFLAGS=-I/usr/local/include', 'LZO_LIBS=-llzo2 -L/usr/local/lib', 'MBEDTLS_CFLAGS=-I/usr/local/Cellar/mbedtls/2.3.0_1/include', 'MBEDTLS_LIBS=-L/usr/local/Cellar/mbedtls/2.3.0_1/lib/ -lmbedtls -lmbedx509 -lmbedcrypto', 'OPENSSL_CFLAGS=-I/usr/local/Cellar/openssl/1.0.2h_1/include/', 'OPENSSL_LIBS=-L/usr/local/Cellar/openssl/1.0.2h_1/lib/ -lcrypto -lssl']: var, value = e.split('=',1) os.environ[var]=value
This method is a bit hackish but works fine.
Buildslaves running out of disk or inodes
The buildslaves consume large amount of diskspace and inodes:
$ df -m|grep -E '(^Filesystem|vda1)' Filesystem 1M-blocks Used Available Use% Mounted on /dev/vda1 9196 4342 4365 50% / $ df -i|grep -E '(^Filesystem|vda1)' Filesystem Inodes IUsed IFree IUse% Mounted on /dev/vda1 606208 274616 331592 46% /
Because buildslave leaves obsolete build directories laying around, removing them can help free up some space and inodes:
$ rm -rf /var/lib/buildbot/slaves/openvpn/build-*
On Ubuntu it is fairly important to remove useless kernels and kernel headers periodically:
$ apt-get autoremove
Buildslave not connecting to buildmaster
The buildslave directory (e.g. /var/lib/buildbot/slaves/openvpn) contains a logfile, twistd.log, which will help you figure out what went wrong. Usually there is an authentication problem, so double-check the buildslave username and password, and edit buildbot.tac as necessary.