Topology subnet doesn't work on FreeBSD 10.0

[garga]

When you run openvpn client on FreeBSD 10 and connect to a server that sets topology to subnet, it doesn't work properly because tun interface is set as POINTTOPOINT when it should be changed to BROADCAST.

Attached patch fix it.

[Gert Döring]

Please provide a log explaining why it is not working today.

Pointtopoint/broadcast is meaningless for a tun interface (it is always "point to point"
from the view of the kernel, as there is no next-hop resolution involved), and the existing code works nicely on FreeBSD up to 9.2 - haven't tested 10.0 yet. The important bit is to have a route pointing the subnet in question into the tun.

To the contrary, setting the interface to BROADCAST breaks IPv6, which is why this was actually reverted about 3 years ago. So we're not going to set IFF_BROADCAST - if needed, we can add code to add a subnet route to the tun interface (in case the ifconfig is not sufficient anymore).

[Eric Crist]

This is working as expected. If you want broadcast traffic, you should be using the TAP adapter. I use OpenVPN extensively on FreeBSD 10 and we have experienced no problems.

[mandree]

downstream bug tracker: ​https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191738

[Gert Döring]

I'm reopening it for now. I want to either have my own freebsd 10 buildslave up and running and test it myself, or want to see a logfile showing failure.

I have seen a failure case with "topology subnet" on an earlier FreeBSD version, but when testing, it turned out that the culprit was (in this case) conflicting routes from previous tests, so on a pristine routing table, everything worked nicely.

In the end, there's two aspects to "broadcast" here:

• whether or not all packets that are matched by ip+netmask are sent by the kernel into the tun interface - this is independent of broadcast/pointtopoint, and purely a matter of "are the routes installed properly". We should get this right.

• whether or not the system tries to do next-hop resolution, specifically neighbour discovery for IPv6 on the tun. Setting the tun to BROADCAST makes FreeBSD run ND for IPv6, which fails for us, because OpenVPN is lacking proper fe80:: handling (the NS packets are just dropped by the server side). This is the reason why the original code that did set IFF_BROADCAST was reverted somewhere in the beta phase before 2.3.0

• most interesting for me to understand whether our horizon is too limited: there could be use cases where programs refuse to "see" a tun interface because said programs "only work on BROADCAST interfaces" - or maybe some sort of broadcast communication is attempted but does not work, because replication of "packets to the broadcast address" to "all clients" doesn't work. This would actually be independent of the IFF_BROADCAST setting (again) as the client openvpn will forward all packets to the server anyway, not having local "intelligence", while the openvpn server has no idea what the OS thinks it's tun interface is configured for - if it comes in via the tun FD, all information left is what is in the IP header.

So, points 1+2 can be summarized as "it should work without, if not, I need to fix the routing setup". Point 3 is "let me see the use case" :-)

[Gert Döring]

So. Installed 10.0p7 and openvpn git master today, tested with topology subnet, announcing a /24 from the server.

This is what I have:

fbsd100.ov$ ifconfig tun0
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500

inet 10.194.3.12 --> 10.194.3.12 netmask 0xffffff00
inet6 fd00:abcd:194:3::100a prefixlen 64

fbsd100.ov$ netstat -rn
10.194.3.0/24 10.194.3.12 UGS 0 148 tun0

and I can see that packets to addresses out of the /24 are properly sent into the tunnel (verified with tcpdump on the server's tun interface). So I think this is good enough for "normal usage".

I'm still curious if there is something I am not aware that needs IFF_BROADCAST. Please let me know.

[garga]

Let me show the case that happened to me and lead me to start looking for a solution. This same scenario works fine on pfSense 2.1 (FreeBSD 8.3) but fails on pfSense 2.2 snapshots (FreeBSD 10-STABLE).

I connect to a server configured with 'topology subnet', It sends me 'ifconfig 172.27.111.2 255.255.255.0' and I end with tun0 set as you mentioned in comment #5. I also have a route for network 172.27.111.0/24 pointing to tun0 and can reach gateway 172.27.111.1 without problems.

But server also sends me 2 networks that I should have access, 172.21.0.0/16 and 172.27.0.0/16, those routes are created in the system using 172.27.111.1 as gw.

Here the issue happens, those routes point to lo0 and I end up with a routing loop. Here is part of my routing table:

172.21.0.0/16 172.27.111.1 UGS lo0
172.27.0.0/16 172.27.111.1 UGS lo0
172.27.111.0/24 172.27.111.2 UGS tun0
172.27.111.2 link#14 UH tun0

Am I doing something wrong here? Let me know if you need more details

[Eric Crist]

This really appears to be a routing issue. Your second route, 172.27.0.0/16 overlaps the route passed in the ifconfig push.

[garga]

It doesn't overlaps it, it contains it, and every OS chooses the most specific match. BTW, this same scenario works fine on FreeBSD 8.3, linux, OSX, Windows.

And on FreeBSD 10, If it's not 'topology subnet' I would end up with a tunnel like '172.27.111.2 -> 172.27.111.1' and the /16 routes would go to tun0 instead of lo0. Same happens after apply this patch and change interface to IFF_BROADCAST and I end up with this items on routing table:

172.21.0.0/16 172.27.111.1 UGS tun0
172.27.0.0/16 172.27.111.1 UGS tun0
172.27.111.0/24 link#14 U tun0

[Gert Döring]

ecrist: /24 beats /16, so this is perfectly fine. (A default route would always overlap everything, and it still works).

garga: this is interesting. I'd need to see the openvpn log. The funny thing is that my test setup does the same thing, pushing an encompassing /16 for the /24, and my automated ping test verifies that this works.

My routing table with the extra route:

Destination Gateway Flags Refs Use Netif Expire
10.194.0.0/16 10.194.3.1 UGS 0 0 tun0
10.194.3.0/24 10.194.3.12 UGS 0 0 tun0
10.194.3.12 link#3 UH 0 0 tun0

and I can ping 10.194.0.1 (a lo0 on the openvpn server) fine.

This is what my OpenVPN log has to say about the routes:

Wed Jul 9 22:11:39 2014 PUSH: Received control message: 'PUSH_REPLY,ifconfig-ipv6 fd00:abcd:194:3::100a/64 fd00:abcd:194:3::1,route 10.194.0.0 255.255.0.0,route-ipv6 fd00:abcd:194::/48,tun-ipv6,route-gateway 10.194.3.1,topology subnet,ping 10,ping-restart 30,ifconfig 10.194.3.12 255.255.255.0'
...
Wed Jul 9 22:11:39 2014 /sbin/ifconfig tun0 10.194.3.12 10.194.3.12 mtu 1500 netmask 255.255.255.0 up
Wed Jul 9 22:11:39 2014 /sbin/route add -net 10.194.3.0 10.194.3.12 255.255.255.0
Wed Jul 9 22:11:39 2014 /sbin/route add -net 10.194.0.0 10.194.3.1 255.255.0.0
...
Wed Jul 9 22:11:39 2014 Initialization Sequence Completed

(leaving off all the IPv6 stuff)

Garga, when you end openvpn, are any 172.* routes still there? OpenVPN used to do this every now and then, leave broken routes around if aborted, but it should properly clean up behind itself nowadays - but if there is anything 172.* before OpenVPN starts, it might confuse "route"

[garga]

cron2, are you testing it on a FreeBSD 10?

PUSH: Received control message: 'PUSH_REPLY,ifconfig-ipv6 xxxx:xxx:11:f111::1000/64 xxxx:xxx:11:f111::1,route 172.27.0.0 255.255.0.0,route-ipv6 xxxx:xxx:11::/48,dhcp-option DNS 172.27.32.5,dhcp-option DNS 172.27.32.6,route 172.21.0.0 255.255.0.0,tun-ipv6,route-gateway 172.27.111.1,topology subnet,ping 10,ping-restart 60,ifconfig 172.27.111.2 255.255.255.0'

[Gert Döring]

FreeBSD 10.0-RELEASE-p7, as I said. OpenVPN git master, but that code should be identical to 2.3.4.

Please show the bits from the log where "/sbin/route add ..." is called (might need "--verb 3"), and please also show "netstat -rn -f inet" before OpenVPN starts, and after it completed initialization.

[garga]

Nevermind, you can close the ticket. It works as expected on stock FreeBSD, it's specific routing issue on pfSense. Thanks!

[Eric Crist]

You're both right - I'm an idiot.

[Gert Döring]

Supposedly it's broken in 11.0-RELEASE now also on "stock FreeBSD". Reopening, revisit the issue.

Seems the "myself" address is no longer logically tied to the tun0 interface but to lo0, so the old trick of pointing a route to "my own address" to mean "go to *this* interface" no longer works.

For tun interfaces, just using -ifp tun0 should be reasonable *if* it's a route intended to go to the tun if (--route has somewhat funny semantics for IPv4) and that would be a fairly simple fix. Not in the next 10 days, though, too busy.

[Gert Döring]

​https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=207831

nicely pinpoints it - it was an change in FreeBSD which broke it, but arguably our approach wasn't "really nice" before that, it just happened to work.

I'll tackle it, should make 2.3.14

[Gert Döring]

this somewhat overlaps with #481 as well

[Gert Döring]

patch is on the list, works for me on FreeBSD 11.0-RELEASE-p3 and 7.4-RELEASE ("most recent and oldest system I have"), both client and server.

Feedback welcome.

Date: Tue, 8 Nov 2016 13:45:06 +0100
From: Gert Doering <gert@…>
To: openvpn-devel@…
Subject: [Openvpn-devel] [PATCH] Repair topology subnet on FreeBSD 11

​https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg12950.html

[garga]

Great! I've imported the fix to pfSense's ports tree [1] and users reported it fixed the issue [2]. Thanks!

[1] ​https://github.com/pfsense/FreeBSD-ports/commit/153999c431c59ac95d71f3214a48d9032a566c58
[2] ​https://redmine.pfsense.org/issues/6883#change-29343

[Gert Döring]

garga, thanks for testing and reporting back - this sped up our review process quite a bit :-)

Here we go...

commit a433b3813d8c38b491d2baa7b433973f2d6cd7c6 (master)
commit 446ef5bda4cdc75d4cb955e274846faff0181fd3 (release/2.3)
Author: Gert Doering
Date: Tue Nov 8 13:45:06 2016 +0100

Repair topology subnet on FreeBSD 11

in-tree, and will show up in 2.3.14 (whenever that's going to hit the shelves)

I'm closing this ticket as there is nothing more to do - if it turns out something is still broken, feel free to reopen (or create a new ticket and point to this one for context).