[FreeBSD] UDP multihome uses wrong IP address if alias is on the same subnet as for main IP address

[Dystopian]

Server:
FreeBSD 10.3-RELEASE i386
OpenVPN 2.4.0 from packages, i386-portbld-freebsd10.3 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD]

ifconfig vtnet0 inet 192.168.1.6/24
ifconfig vtnet0 inet 192.168.1.7/32 alias # note the same subnet

openvpn --multihome --cipher none --auth none --secret key.txt --dev tun --topology p2p --ifconfig 10.0.0.1 10.0.0.2

Client 192.168.1.5:
openvpn --cipher none --auth none --secret key.txt --dev tun --topology p2p --ifconfig 10.0.0.2 10.0.0.1 --remote 192.168.1.7

tcpdump -nti vtnet0
IP 192.168.1.5.1194 > 192.168.1.7.1194: UDP, length 25
IP 192.168.1.6.1194 > 192.168.1.5.1194: UDP, length 164

Tested also on 10.3-RELEASE amd64 server, the same result.
If using different subnet for alias (e.g. 192.168.0.*), it's OK.

The problem is fixed if "--proto udp4" option is used. The main difference in server log then is absence of such line:
Could not determine IPv4/IPv6 protocol. Using AF_INET6

[Gert Döring]

There should be something in the server log about the IP address seen in the incoming packet - so please show the parts of the server log with "--verb 3" when the connection comes in.

I have tested --multihome with v4 and v6 on FreeBSD, but it's possible that it breaks for v4-mapped addresses on dual-stacked v6 sockets (which I have not tested) - we've seen that on Linux, and it turned out to be a linux kernel omission (the ancilliary data path was completely missing for the v4-mapped v6-socket case) - the fact that it works if you force IPv4 is an indication. But let's see the log first.

[Dystopian]

--verb 5
us=437012 Peer Connection Initiated with [AF_INET6]::ffff:192.168.1.5:1194 (via ::ffff:192.168.1.7%vtnet0)
us=437088 Initialization Sequence Completed
WRW

[Gert Döring]

OK, so the receiving side code path (both FreeBSD side and OpenVPN side) is good - that's at least something :-)

For the sending side (which is what ensures that the UDP source is also set to that value) - anything interesting in the log, like an error from sendmsg()? Or does everything look nice, except that it's not doing the right thing?

[Dystopian]

Nothing from sendmsg.
verb 5 doesn't show anything about sending.
Here verb 11 (I hope it's what you need):

us=100810 SENT OCC_REPLY
us=100901 ENCRYPT TO: 00000008 58aacb2f 287f346b d4ef7a81 2d56b8d3 afc5459c 0156342c 6465762[more...]
us=100932 PO_CTL rwflags=0x0003 ev=4 arg=0x080e53cc
us=100960 PO_CTL rwflags=0x0000 ev=3 arg=0x080e4674
us=100993 I/O WAIT Tr|Tw|SR|SW [1/231634]
us=101085 PO_WAIT[0,0] fd=4 rev=0x00000004 rwflags=0x0002 arg=0x080e53cc
us=101116  event_wait returned 1
us=101145 I/O WAIT status=0x0002
us=101276 UDPv6 WRITE [164] to [AF_INET6]::ffff:192.168.1.5:1194 (via ::ffff:192.168.1.7%vtnet0):  DATA 00000008 58aacb2f 287f346b d4ef7a81 2d56b8d3 afc5459c 0156342c 6465762[more...]
us=101421 UDPv6 write returned 164
us=101471 PO_CTL rwflags=0x0001 ev=4 arg=0x080e53cc
us=101507 PO_CTL rwflags=0x0001 ev=3 arg=0x080e4674
us=101540 I/O WAIT TR|Tw|SR|Sw [1/231634]

[Dystopian]

I don't know how to subscribe to the ticket and I didn't receive any emails on new comments.
If you need more info from me send me email please.

[Gert Döring]

anyone who adds to a trac ticket is "auto-subscribed" - you didn't hear anything because I did not find time to investigate what is happening there.

From your logs, everything looks reasonable - we figure out what the right address should be, and there are no errors on sendmsg(). So it's not something straightforward... - debug time.

[Dystopian]

Don't think I complain on slow ticket resolving, I just meant I received only 1 email from ticket beginning - after my previous comment. I would like to receive emails after all comments except mine.

[Gert Döring]

So, that topic came back, and I forgot about this ticket in the meantime - the other one is #1057

The problem here is that --proto udp will create a dual-stack IPv6 socket, which can also handle IPv4 addresses but the code path in the kernel to set the source address for IPv4 packets leaving via such a socket was just missing. Like, FreeBSD had no code to do that, so picked the first interface IP.

For --proto udp4, it's a different kernel code path, so it works.

For IPv6 addresses, --proto udp and --multihome also works.

In the context of #1057, a FreeBSD bug was opened, addressed, tested and merged - so 12.2-RELEASE will have this fixed.

So - sorry for ignoring this ticket for such a long time, but in the end, things got debugged and fixed (and, unfortunately, there is nothing in OpenVPN to work around this, except use proto udp4, as you already discovered). Closing this ticket now.