CI for DCO Linux

Distributing an out-of-tree kernel module means signing up to support a wide variety of kernels. An enterprise distro freezes one for years and backports fixes into it the whole time; a rolling release ships something new every week; an LTS sits somewhere in between. Distros carry these very different kernels and patch them on their own schedule, without asking anyone. A stable-tree update or an enterprise backport can quietly change an assumption the module relied on, and the first sign of it is a bug report.

So the OpenVPN kernel modules need more than a normal userspace CI job. A useful test has to build against real distro kernel headers, boot that kernel, load the module, and either run the module selftests or at least prove the module still compiles on the supported platforms.

That creates an awkward constraint: free GitHub-hosted runners are only provided as Ubuntu, macOS, or Windows machines, while the module has to work across Debian, Ubuntu, Fedora, RHEL, and openSUSE kernels. The CI therefore cannot simply "use a Fedora runner" or "use a RHEL runner". (Red Hat Enterprise Linux runner images entered public preview in mid-2026, but only on the larger paid runners.) Instead, the Ubuntu runner becomes a build host that generates a target distro rootfs, boots it in a nested VM, and runs the actual module workload inside that guest.

The result is kmod-ci, a reusable GitHub Actions workflow for out-of-tree OpenVPN kernel modules. Caller repositories such as ovpn-backports and ovpn-dco only provide the small module-specific script to run inside the guest. The shared CI repository owns the distro matrix, rootfs generation, virtme-ng boot logic, RHEL handling, and scheduled-run optimization.

High-level flow

Each matrix job starts on a standard ubuntu-24.04 or ubuntu-24.04-arm64 GitHub runner. The runner installs the host-side tools needed to build root filesystems and boot virtual machines: mmdebstrap, dnf, qemu, virtme-ng, virtiofsd, and related utilities.

The job then checks out two repositories:

• the caller repository: the module repository being tested, copied into the guest as /repo.
• the shared CI repository: it provides the scripts that know how to generate root filesystems and boot the guest.

For each distro target, the workflow builds a rootfs using that distro's normal package repositories. Debian and Ubuntu are generated with mmdebstrap. Fedora, RHEL, AlmaLinux, and openSUSE targets are generated with dnf --installroot. The generated rootfs includes the distro kernel, kernel headers, compiler toolchain, module build dependencies, and the small set of userspace tools needed by the module tests.

For RPM-based targets, the shared CI also carries the small repository definitions needed to bootstrap the installroot. They are based on the official repo files shipped by each distribution, with only the adjustments needed to use them from an Ubuntu GitHub runner.

Once the rootfs exists, the workflow boots it with virtme-ng. Inside the guest, the caller-provided script runs as root. For ovpn-backports on x86_64, that script builds the module and runs the ovpn kselftests. For ovpn-dco and ovpn-backports on arm64, the current useful signal is primarily whether the module still builds across the supported kernel matrix.

Why boot the distro kernel?

The important part is that the module is not only compiled on Ubuntu. It is compiled and loaded against the kernel shipped by the target distribution.

That matters because enterprise distributions routinely carry large backport sets. A RHEL 9 kernel may report a 5.14 base version, but its APIs do not behave exactly like upstream Linux 5.14. The same is true, to varying degrees, for SUSE and other long-term distro kernels. Testing only upstream version numbers would miss the real compatibility surface.

The CI therefore treats the distro kernel package as the source of truth. It installs the kernel and matching headers from the distro repositories, boots that kernel, and runs the module workload there.

Current distro matrix

The exact kernel release is not hardcoded in the workflow. It moves whenever the distribution updates the corresponding kernel package. At the time of writing, the full selftest matrix covers this kernel range:

RHEL support

RHEL is the one target family that cannot be generated from public repository files alone. The workflow uses Red Hat subscription-manager inside a temporary Red Hat Universal Base Image (UBI) container to authenticate against Red Hat CDN and enable the required repositories.

AlmaLinux remains supported as a convenient EL-family target, and it was very useful while the CI was being brought up because it made iteration quick and did not require credentials. For this project, though, real RHEL is the better signal. RHEL kernel updates are exactly the updates that often matter to users running OpenVPN in enterprise environments, and clones only see those changes after Red Hat has already shipped them.

The caller repository provides two GitHub Actions secrets:

• RHEL_ORG_ID
• RHEL_ACTIVATION_KEY

Those secrets are passed only to the RHEL rootfs generation step. The UBI container registers with RHSM, enables BaseOS, AppStream, and CodeReady Builder, generates the redhat.repo file, and uses that to build the installroot.

The container is run privileged because the RPM rootfs generation path mounts /dev, /proc, /sys, and /run into the installroot. Kernel package scriptlets expect something closer to a normal running system, especially when installing kernel packages and running dracut or kernel-install hooks.

The registration is cleaned up on exit. This avoids accumulating RHSM consumer registrations on every matrix run, including failing runs. The generated rootfs is also scrubbed defensively so RHSM identity and entitlement state are not carried forward into the VM artifact.

virtme-ng and filesystem transports

The guest is booted with virtme-ng because it is much lighter than maintaining full disk images for every distro. The workflow generates a directory rootfs, asks virtme-ng to boot the target kernel, and exposes the caller repository to the guest. It also aligns with NIPA, the netdev CI system, which uses virtme-ng for kernel test boots as well.

Most targets use virtiofs, which is fast enough for repeated module builds and selftests. Older kernels can be more difficult. Debian 10, in particular, uses a 4.19 kernel and has to fall back to the legacy 9p path in this setup. That target is noticeably slower, but still useful because it covers the oldest kernel generation the module is expected to support.

The other important performance detail is KVM. GitHub's x86_64 Linux runners currently expose /dev/kvm, so nested virtualization works and the guest boots are quick. The guest runner script still probes for /dev/kvm before booting. If it is missing, it adds --disable-kvm and continues with software emulation rather than failing outright. That fallback is a safety net: GitHub documents nested virtualization on hosted runners as "experimental and done at your own risk," with no guarantees on stability or performance. In fact, on arm64, KVM is not exposed so we resort to software emulation which is not only way slower but also instable (for example it occasionally segfaults while building or installing the module). For this reason arm64 targets are kept minimal and manual-dispatched only.

RHEL kernels also forced work in this area because they do not provide 9p at all, and virtme-ng historically mounted some helper exports over 9p. That work became virtme-ng PR #472, which removes the remaining hardcoded 9p helper exports and uses virtiofs instead when virtiofs is available.

Addidionatlly, some older kernels failed external-rootfs virtiofs boots when virtiofsd was started with --posix-acl. virtme-ng PR #482 added --no-root-posix-acl to explicitly control this behavior.

Finally, newer arm64 distro kernels are no longer plain Image files from QEMU's point of view. Fedora 44 and recent Ubuntu development releases ship EFI zboot images, and some Ubuntu kernels add another signed PE wrapper around the bootable payload. virtme-ng PR #483 added support for arm64 kernel images normalization before passing them to QEMU.

Scheduled runs

The workflow can be triggered manually, but the x86_64 matrix is also designed to run nightly.

That is useful because the module repository may not change every day, while distro kernels do. A RHEL or openSUSE kernel update can break the module even if no OpenVPN code changed. Scheduled CI catches that kind of regression soon after the distro publishes the update.

To keep scheduled runs from wasting time, the workflow records a small success marker keyed by the caller repository revision, the distro target, and the kernel release installed in the generated rootfs. On a scheduled run, the rootfs is still generated first, because that is the reliable way to know which kernel the distro currently serves. If the same repo revision has already passed with that same target kernel, the expensive guest workload is skipped.

This keeps the behavior simple and correct. It does not try to predict repository state with a separate metadata parser. The package manager remains the source of truth.

Runtime

With the ovpn-backports selftest workload, most targets complete in roughly seven to eleven minutes. Debian 10 is the expected outlier: because it uses the older 9p path, it takes about fifteen minutes, and therefore determines the total runtime of the full parallel matrix.

The x86_64 ovpn-dco workload is shorter because it does not run the kselftests. The Ubuntu and Debian jobs complete in about two minutes, and the RHEL jobs complete in roughly three to five minutes.

The arm64 matrix, on the other hand, can take up to twenty minutes despite being compile-only, because of the limitations of software emulation.

Shared workflow model

The useful boundary is that the shared CI owns the infrastructure, while each module repository owns only the command that should run inside the guest.

That keeps the caller workflows small. A repository can opt into the matrix by calling the reusable workflow and passing:

• the cache prefix,
• the guest script path,
• optionally a preparation command,
• optionally a custom distro list.

The same mechanism can support both ovpn-backports and ovpn-dco without duplicating the rootfs and virtme-ng machinery in each repository.

It also keeps distro-specific quirks contained. Debian archive handling, Ubuntu keyrings, RHEL subscription setup, openSUSE repository files, old-kernel 9p fallback, and RPM scriptlet behavior are all handled in one place.

Tradeoffs

This is not as fast as a native runner per distribution would be, but those runners do not exist on GitHub-hosted Actions. It is also not as hermetic as prebuilt qcow images, but avoiding checked-in or externally hosted VM images makes the CI self-contained and easier to reproduce.

The rootfs is generated every time before the scheduled cache decision. That is intentionally a little wasteful. The alternative would be duplicating package-manager logic just to predict whether the kernel changed, and that would be more fragile than letting apt or dnf answer the question directly.

Some distro rough edges remain. Debian 10 is slow because it falls back to 9p. openSUSE has required explicit key handling and has shown transient failures caused by repo unreachability. RHEL requires credentials and a temporary RHSM registration.

Current status

The system now runs nightly across the default x86_64 matrix and gives the OpenVPN kernel modules a real distro-kernel compatibility signal on ordinary GitHub-hosted runners.

Tested modules:

• ovpn: https://github.com/OpenVPN/ovpn-backports/actions
• ovpn-dco-v2: https://github.com/OpenVPN/ovpn-dco/actions