spurious failures with IPv6 over TAP on FreeBSD 12 (ND6_IFF_IFDISABLED)

[Gert Döring]

Setting up a new testbed with the clients run on FreeBSD 12 (12.0-RELEASE-p11) I ran into spurious IPv6 failures in the TAP tests.

In all those cases, the tap interface config showed

tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:2d:62:5f:00
        inet6 fe80::2bd:2dff:fe62:5f00%tap0 prefixlen 64 tentative scopeid 0x3
        inet6 fd00:abcd:204:4::1001 prefixlen 64 tentative
        inet 10.204.4.3 netmask 0xffffff00 broadcast 10.204.4.255
        groups: tap
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 65794

all IPv6 addresses are flagged as "tentative" (and stay so), and the "IFDISABLED" flag is set. In all cases that work, this flag is not set.

This is not dependent on OpenVPN version (it happens with 2.4.8 and git master) nor config (if you run the same test 8 times in a row, most times it will succeed and sometimes it fails).

It is very likely timing dependent, as it seems to happen more often on a FreeBSD 11.3 VM which is run on a different (faster) host...

The main intention of this ticket is "record the fact so it can be found with search engines". I think it's a FreeBSD bug, but have no time right now to find people that can look into it.

One possible workaround is to use an up script

script-security 2
up /usr/local/etc/openvpn/inet6-no-ifdisabled.sh

which contains

#!/bin/sh
IF=$1

( sleep 2 ; ifconfig $IF inet6 -ifdisabled ) &

(the sleep 2 is needed) - with this script, the problem completely goes away for all cases.

[Gert Döring]

Unfortunately, this is a race condition somewhere - so even if I add an explicit "ifconfig inet6 -ifdisable" to OpenVPN, FreeBSD will turn it back on behind our back.

This is from a test run, running "sleep 1 ; ifconfig tun0" in a loop...

tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::250:56ff:fe9c:dffb%tun0 prefixlen 64 tentative scopeid 0x3
        inet6 fd00:abcd:204:2::1000 prefixlen 64 tentative
        inet 10.204.2.6 --> 10.204.2.5 netmask 0xffffffff
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 34820
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet6 fe80::250:56ff:fe9c:dffb%tun0 prefixlen 64 tentative scopeid 0x3
        inet6 fd00:abcd:204:2::1000 prefixlen 64 tentative
        inet 10.204.2.6 --> 10.204.2.5 netmask 0xffffffff
        groups: tun
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 34820

As with the --up script, there has to be a sleep() between "ifconfig inet6 $addr" and "ifconfig inet6 -ifdisable".

As the ticket title says "FreeBSD 11" I have re-tested this, but cannot reproduce the problem there (without the workaround, running master as of today). It never says "I tested and saw it there", so I must have been confused :-) ("man ifconfig" mentions that flag, but I've never seen it get set on its own).

Patch is trivial, if ugly...

--- a/src/openvpn/tun.c
+++ b/src/openvpn/tun.c
@@ -1038,6 +1038,23 @@ do_ifconfig_ipv6(struct tuntap *tt, const char *ifname, int tun_mtu,
     openvpn_execve_check(&argv, es, S_FATAL,
                          "generic BSD ifconfig inet6 failed");
 
+#if defined(TARGET_FREEBSD) && __FreeBSD_version >= 1200000
+    /* FreeBSD 12 grew some sort of race condition where sometimes
+     * the IPv6 side of a tun interface is stuck in "IFDISABLED"
+     * in "ifconfig tun0" output, and the assigned IPv6 address is not
+     * working ("tentative").  Workaround in here, for now.
+     *
+     * the sleep is needed because there is a race condition, so turning
+     * it off immediately can lead to "kernel turns it on again, stuck!!"
+     */
+    sleep(1);
+    argv_printf(&argv, "%s %s inet6 -ifdisabled", IFCONFIG_PATH, ifname);
+    argv_msg(M_INFO, &argv);
+
+    openvpn_execve_check(&argv, es, S_FATAL,
+                         "FreeBSD BSD 'ifconfig inet6 -ifdisabled' failed");
+#endif

[Gert Döring]

Patch on the list, needs to go into 2.4 and 2.5

[Gert Döring]

FreeBSD PR opened... ​https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248172

[Gert Döring]

The FreeBSD folks have pointed me to /etc/pccard_ether and ipv6_activate_all_interfaces="YES" - which helps understand the cause of the problem.

Unfortunately, they see this as "works as designed" (if you want IPv6, enable it in rc.conf) while I see this from an OpenVPN user point of view, who might not have or want IPv6 "in his normal environment" but *does* want IPv6 in his VPN, without wanting to configure anything outside of OpenVPN...

Until we can find a better agreement, I am merging this. The sleep(1) looks race'y, but since the critical time window is very short, this is good enough (it's hard enough to purposely hit the problem - it needs specifics, like "disk activity" and "slow VM storage").

I am closing this one for now. If I find a better workaround, I might reopen and comment again.

commit 5e19cc2c1bf22d44f10dc585ba69d48951dc7a86 (master)
commit 6dac3342d851995bdee75db4b52cfb3a3f9232e3 (release/2.5)
Author: Gert Doering
Date: Thu Jul 23 14:19:49 2020 +0200

Workaround FreeBSD 12+ race condition on tun/tap open with IPv6.

[Gert Döring]

FreeBSD has a fix now in their tree

​https://reviews.freebsd.org/D27324
​https://reviews.freebsd.org/rS368031

which fixes this in a much nicer way (no more race condition). So the sleep(1) can go, "if it's the right FreeBSD version".

Not sure yet which releases this will hit - maybe 12.3-RELEASE

[Gert Döring]

Just re-checked this. 12.2-RELEASE-p8 still has the offending _ipv6_opts="ifdisabled" in /etc/network.subr, while it's fixed in 13.0-RELEASE.

Waiting for 12.3-RELEASE and for 11.4+12.2 to become unsupported - then we can remove the workaround in OpenVPN's code again.

Bumping to 2.5.4 milestone as a reminder "check on FreeBSD again".

[Gert Döring]

Still present in 12.3, gone from 12.4, 13.1 - so, will prepare a patch that removes the workaround again for sufficiently recent FreeBSD versions.

[Gert Döring]

Workaround has been deactivated for "12.4 and up" (so technically 13.0 would now be "broken" again, but 13.0 is no longer supported and 13.1 is fixed).

commit 16d7f2cd4d904274580b2f031e92dde2f7f260c9 (master)
commit d9906d1523a8ed46fce81cb8d559279e345bf849 (release/2.6)
Author: Gert Doering
Date: Sat Jan 7 17:25:58 2023 +0100

Undo FreeBSD 12.x workaround on IPv6 ifconfig for 12.4 and up

Putting the milestone now to 2.7.0 to fully remove this block.