# CI for DCO Linux

Distributing an out-of-tree kernel module means signing up to support a wide
variety of kernels. An enterprise distro freezes one for years and backports
fixes into it the whole time; a rolling release ships something new every week;
an LTS sits somewhere in between. Distros carry these very different kernels
and patch them on their own schedule, without asking anyone. A stable-tree
update or an enterprise backport can quietly change an assumption the module
relied on, and the first sign of it is a bug report.

So the OpenVPN kernel modules need more than a normal userspace CI job. A
useful test has to build against real distro kernel headers, boot that kernel,
load the module, and either run the module selftests or at least prove the
module still compiles on the supported platforms.

That creates an awkward constraint: free GitHub-hosted runners are only provided
as Ubuntu, macOS, or Windows machines, while the module has to work across
Debian, Ubuntu, Fedora, RHEL, and openSUSE kernels. The CI therefore cannot
simply "use a Fedora runner" or "use a RHEL runner". (Red Hat Enterprise Linux
runner images entered [public preview](https://github.blog/changelog/2026-06-25-red-hat-enterprise-linux-runner-images-are-now-in-public-preview/)
in mid-2026, but only on the larger paid runners.) Instead, the Ubuntu runner
becomes a build host that generates a target distro rootfs, boots it in a nested
VM, and runs the actual module workload inside that guest.

The result is [kmod-ci](https://github.com/mandelbitdev/kmod-ci), a reusable
GitHub Actions workflow for out-of-tree OpenVPN kernel modules. Caller
repositories such as
[`ovpn-backports`](https://github.com/OpenVPN/ovpn-backports) and
[`ovpn-dco`](https://github.com/OpenVPN/ovpn-dco) only provide the small
module-specific script to run inside the guest. The shared CI repository owns
the distro matrix, rootfs generation, virtme-ng boot logic, RHEL handling, and
scheduled-run optimization.

## High-level flow

Each matrix job starts on a standard `ubuntu-24.04` or `ubuntu-24.04-arm64`
GitHub runner. The runner installs the host-side tools needed to build root
filesystems and boot virtual machines: `mmdebstrap`, `dnf`, `qemu`,
`virtme-ng`, `virtiofsd`, and related utilities.

The job then checks out two repositories:

- the caller repository: the module repository being tested, copied into the
  guest as `/repo`.
- the shared CI repository: it provides the scripts that know how to generate
  root filesystems and boot the guest.

For each distro target, the workflow builds a rootfs using that distro's normal
package repositories. Debian and Ubuntu are generated with `mmdebstrap`.
Fedora, RHEL, AlmaLinux, and openSUSE targets are generated with `dnf
--installroot`. The generated rootfs includes the distro kernel, kernel
headers, compiler toolchain, module build dependencies, and the small set of
userspace tools needed by the module tests.

For RPM-based targets, the shared CI also carries the small repository
definitions needed to bootstrap the installroot. They are based on the official
repo files shipped by each distribution, with only the adjustments needed to use
them from an Ubuntu GitHub runner.

Once the rootfs exists, the workflow boots it with virtme-ng. Inside the guest,
the caller-provided script runs as root. For `ovpn-backports` on x86_64, that
script builds the module and runs the ovpn kselftests. For `ovpn-dco` and
`ovpn-backports` on arm64, the current useful signal is primarily whether the
module still builds across the supported kernel matrix.

## Why boot the distro kernel?

The important part is that the module is not only compiled on Ubuntu. It is
compiled and loaded against the kernel shipped by the target distribution.

That matters because enterprise distributions routinely carry large backport
sets. A RHEL 9 kernel may report a 5.14 base version, but its APIs do not
behave exactly like upstream Linux 5.14. The same is true, to varying degrees,
for SUSE and other long-term distro kernels. Testing only upstream version
numbers would miss the real compatibility surface.

The CI therefore treats the distro kernel package as the source of truth. It
installs the kernel and matching headers from the distro repositories, boots
that kernel, and runs the module workload there.

## Current distro matrix

The exact kernel release is not hardcoded in the workflow. It moves whenever
the distribution updates the corresponding kernel package. At the time of
writing, the full selftest matrix covers this kernel range:

| Target | Kernel family | Notes |
| --- | --- | --- |
| Debian 10 | 4.19 | old LTS coverage |
| Debian 11 | 5.10 | LTS coverage |
| Debian 12 | 6.1 | LTS coverage |
| Debian 13 | 6.12 | LTS coverage |
| Fedora 44 | 7.1 | fast-moving Fedora target |
| openSUSE Leap 15.6 | 6.4 | enterprise-style openSUSE target |
| openSUSE Leap 16.0 | 6.12 | enterprise-style openSUSE target |
| openSUSE Tumbleweed | 7.1 | rolling release |
| RHEL 8 | 4.18 | enterprise LTS kernel |
| RHEL 9 | 5.14 | enterprise LTS kernel |
| RHEL 10 | 6.12 | enterprise LTS kernel |
| Ubuntu 20.04 | 5.4 | LTS coverage |
| Ubuntu 22.04 | 5.15 | LTS coverage |
| Ubuntu 24.04 | 6.8 | LTS coverage |
| Ubuntu 25.10 | 6.17 | interim Ubuntu target |
| Ubuntu 26.04 | 7.0 | LTS coverage |

## RHEL support

RHEL is the one target family that cannot be generated from public repository
files alone. The workflow uses Red Hat `subscription-manager` inside a
temporary [Red Hat Universal Base Image
(UBI)](https://catalog.redhat.com/en/software/base-images) container to
authenticate against Red Hat CDN and enable the required repositories.

AlmaLinux remains supported as a convenient EL-family target, and it was very
useful while the CI was being brought up because it made iteration quick and
did not require credentials. For this project, though, real RHEL is the better
signal. RHEL kernel updates are exactly the updates that often matter to users
running OpenVPN in enterprise environments, and clones only see those changes
after Red Hat has already shipped them.

The caller repository provides two GitHub Actions secrets:

- `RHEL_ORG_ID`
- `RHEL_ACTIVATION_KEY`

Those secrets are passed only to the RHEL rootfs generation step. The UBI
container registers with RHSM, enables BaseOS, AppStream, and CodeReady
Builder, generates the `redhat.repo` file, and uses that to build the
installroot.

The container is run privileged because the RPM rootfs generation path mounts
`/dev`, `/proc`, `/sys`, and `/run` into the installroot. Kernel package
scriptlets expect something closer to a normal running system, especially when
installing kernel packages and running dracut or kernel-install hooks.

The registration is cleaned up on exit. This avoids accumulating RHSM consumer
registrations on every matrix run, including failing runs. The generated rootfs
is also scrubbed defensively so RHSM identity and entitlement state are not
carried forward into the VM artifact.

## virtme-ng and filesystem transports

The guest is booted with virtme-ng because it is much lighter than maintaining
full disk images for every distro. The workflow generates a directory rootfs,
asks virtme-ng to boot the target kernel, and exposes the caller repository to
the guest. It also aligns with [NIPA](https://github.com/linux-netdev/nipa),
the netdev CI system, which uses virtme-ng for kernel test boots as well.

Most targets use virtiofs, which is fast enough for repeated module builds and
selftests. Older kernels can be more difficult. Debian 10, in particular, uses
a 4.19 kernel and has to fall back to the legacy 9p path in this setup. That
target is noticeably slower, but still useful because it covers the oldest
kernel generation the module is expected to support.

The other important performance detail is KVM. GitHub's x86_64 Linux runners currently
expose `/dev/kvm`, so nested virtualization works and the guest boots are quick.
The guest runner script still probes for `/dev/kvm` before booting. If it is
missing, it adds `--disable-kvm` and continues with software emulation rather
than failing outright. That fallback is a safety net: GitHub documents nested
virtualization on hosted runners as
["experimental and done at your own risk," with no guarantees on stability or performance](https://docs.github.com/en/actions/concepts/runners/github-hosted-runners#runner-images).
In fact, on arm64, KVM is not exposed so we resort to software emulation which
is not only way slower but also instable (for example it occasionally segfaults
while building or installing the module). For this reason arm64 targets are
kept minimal and manual-dispatched only.

RHEL kernels also forced work in this area because they do not provide 9p at
all, and virtme-ng historically mounted some helper exports over 9p. That
work became
[`virtme-ng` PR #472](https://github.com/arighi/virtme-ng/pull/472), which
removes the remaining hardcoded 9p helper exports and uses virtiofs instead
when virtiofs is available.

Addidionatlly, some older kernels failed external-rootfs virtiofs boots when
`virtiofsd` was started with `--posix-acl`. [`virtme-ng` PR
#482](https://github.com/arighi/virtme-ng/pull/482) added `--no-root-posix-acl`
to explicitly control this behavior.

Finally, newer arm64 distro kernels are no longer plain Image files from QEMU's
point of view. Fedora 44 and recent Ubuntu development releases ship EFI zboot
images, and some Ubuntu kernels add another signed PE wrapper around the
bootable payload. [`virtme-ng` PR
#483](https://github.com/arighi/virtme-ng/pull/483) added support for arm64
kernel images normalization before passing them to QEMU.

## Scheduled runs

The workflow can be triggered manually, but the x86_64 matrix is also designed
to run nightly.

That is useful because the module repository may not change every day, while
distro kernels do. A RHEL or openSUSE kernel update can break the module even
if no OpenVPN code changed. Scheduled CI catches that kind of regression soon
after the distro publishes the update.

To keep scheduled runs from wasting time, the workflow records a small success
marker keyed by the caller repository revision, the distro target, and the
kernel release installed in the generated rootfs. On a scheduled run, the
rootfs is still generated first, because that is the reliable way to know which
kernel the distro currently serves. If the same repo revision has already
passed with that same target kernel, the expensive guest workload is skipped.

This keeps the behavior simple and correct. It does not try to predict
repository state with a separate metadata parser. The package manager remains
the source of truth.

## Runtime

With the `ovpn-backports` selftest workload, most targets complete in roughly
seven to eleven minutes. Debian 10 is the expected outlier: because it uses the
older 9p path, it takes about fifteen minutes, and therefore determines the
total runtime of the full parallel matrix.

The x86_64 `ovpn-dco` workload is shorter because it does not run the
kselftests. The Ubuntu and Debian jobs complete in about two minutes, and the
RHEL jobs complete in roughly three to five minutes.

The arm64 matrix, on the other hand, can take up to twenty minutes despite
being compile-only, because of the limitations of software emulation.

## Shared workflow model

The useful boundary is that the shared CI owns the infrastructure, while each
module repository owns only the command that should run inside the guest.

That keeps the caller workflows small. A repository can opt into the matrix by
calling the reusable workflow and passing:

- the cache prefix,
- the guest script path,
- optionally a preparation command,
- optionally a custom distro list.

The same mechanism can support both `ovpn-backports` and `ovpn-dco` without
duplicating the rootfs and virtme-ng machinery in each repository.

It also keeps distro-specific quirks contained. Debian archive handling, Ubuntu
keyrings, RHEL subscription setup, openSUSE repository files, old-kernel 9p
fallback, and RPM scriptlet behavior are all handled in one place.

## Tradeoffs

This is not as fast as a native runner per distribution would be, but those
runners do not exist on GitHub-hosted Actions. It is also not as hermetic as
prebuilt qcow images, but avoiding checked-in or externally hosted VM images
makes the CI self-contained and easier to reproduce.

The rootfs is generated every time before the scheduled cache decision. That is
intentionally a little wasteful. The alternative would be duplicating
package-manager logic just to predict whether the kernel changed, and that
would be more fragile than letting apt or dnf answer the question directly.

Some distro rough edges remain. Debian 10 is slow because it falls back to 9p.
openSUSE has required explicit key handling and has shown transient failures
caused by repo unreachability. RHEL requires credentials and a temporary RHSM
registration.

## Current status

The system now runs nightly across the default x86_64 matrix and gives the
OpenVPN kernel modules a real distro-kernel compatibility signal on ordinary
GitHub-hosted runners.

Tested modules:
- ovpn: https://github.com/OpenVPN/ovpn-backports/actions
- ovpn-dco-v2: https://github.com/OpenVPN/ovpn-dco/actions