[FreeBSD] UDP multihome uses wrong IP address if alias is on the same subnet as for main IP address

[Dystopian]

Server:
FreeBSD 10.3-RELEASE i386
OpenVPN 2.4.0 from packages, i386-portbld-freebsd10.3 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD]

ifconfig vtnet0 inet 192.168.1.6/24
ifconfig vtnet0 inet 192.168.1.7/32 alias # note the same subnet

openvpn --multihome --cipher none --auth none --secret key.txt --dev tun --topology p2p --ifconfig 10.0.0.1 10.0.0.2

Client 192.168.1.5:
openvpn --cipher none --auth none --secret key.txt --dev tun --topology p2p --ifconfig 10.0.0.2 10.0.0.1 --remote 192.168.1.7

tcpdump -nti vtnet0
IP 192.168.1.5.1194 > 192.168.1.7.1194: UDP, length 25
IP 192.168.1.6.1194 > 192.168.1.5.1194: UDP, length 164

Tested also on 10.3-RELEASE amd64 server, the same result.
If using different subnet for alias (e.g. 192.168.0.*), it's OK.

The problem is fixed if "--proto udp4" option is used. The main difference in server log then is absence of such line:
Could not determine IPv4/IPv6 protocol. Using AF_INET6

[Gert Döring]

There should be something in the server log about the IP address seen in the incoming packet - so please show the parts of the server log with "--verb 3" when the connection comes in.

I have tested --multihome with v4 and v6 on FreeBSD, but it's possible that it breaks for v4-mapped addresses on dual-stacked v6 sockets (which I have not tested) - we've seen that on Linux, and it turned out to be a linux kernel omission (the ancilliary data path was completely missing for the v4-mapped v6-socket case) - the fact that it works if you force IPv4 is an indication. But let's see the log first.

[Dystopian]

--verb 5
us=437012 Peer Connection Initiated with [AF_INET6]::ffff:192.168.1.5:1194 (via ::ffff:192.168.1.7%vtnet0)
us=437088 Initialization Sequence Completed
WRW

[Gert Döring]

OK, so the receiving side code path (both FreeBSD side and OpenVPN side) is good - that's at least something :-)

For the sending side (which is what ensures that the UDP source is also set to that value) - anything interesting in the log, like an error from sendmsg()? Or does everything look nice, except that it's not doing the right thing?

[Dystopian]

Nothing from sendmsg.
verb 5 doesn't show anything about sending.
Here verb 11 (I hope it's what you need):

us=100810 SENT OCC_REPLY
us=100901 ENCRYPT TO: 00000008 58aacb2f 287f346b d4ef7a81 2d56b8d3 afc5459c 0156342c 6465762[more...]
us=100932 PO_CTL rwflags=0x0003 ev=4 arg=0x080e53cc
us=100960 PO_CTL rwflags=0x0000 ev=3 arg=0x080e4674
us=100993 I/O WAIT Tr|Tw|SR|SW [1/231634]
us=101085 PO_WAIT[0,0] fd=4 rev=0x00000004 rwflags=0x0002 arg=0x080e53cc
us=101116  event_wait returned 1
us=101145 I/O WAIT status=0x0002
us=101276 UDPv6 WRITE [164] to [AF_INET6]::ffff:192.168.1.5:1194 (via ::ffff:192.168.1.7%vtnet0):  DATA 00000008 58aacb2f 287f346b d4ef7a81 2d56b8d3 afc5459c 0156342c 6465762[more...]
us=101421 UDPv6 write returned 164
us=101471 PO_CTL rwflags=0x0001 ev=4 arg=0x080e53cc
us=101507 PO_CTL rwflags=0x0001 ev=3 arg=0x080e4674
us=101540 I/O WAIT TR|Tw|SR|Sw [1/231634]

[Dystopian]

I don't know how to subscribe to the ticket and I didn't receive any emails on new comments.
If you need more info from me send me email please.

[Gert Döring]

anyone who adds to a trac ticket is "auto-subscribed" - you didn't hear anything because I did not find time to investigate what is happening there.

From your logs, everything looks reasonable - we figure out what the right address should be, and there are no errors on sendmsg(). So it's not something straightforward... - debug time.

[Dystopian]

Don't think I complain on slow ticket resolving, I just meant I received only 1 email from ticket beginning - after my previous comment. I would like to receive emails after all comments except mine.

[Gert Döring]

So, that topic came back, and I forgot about this ticket in the meantime - the other one is #1057

The problem here is that --proto udp will create a dual-stack IPv6 socket, which can also handle IPv4 addresses but the code path in the kernel to set the source address for IPv4 packets leaving via such a socket was just missing. Like, FreeBSD had no code to do that, so picked the first interface IP.

For --proto udp4 , it's a different kernel code path, so it works.

For IPv6 addresses, --proto udp and --multihome also works.

In the context of #1057 , a FreeBSD bug was opened, addressed, tested and merged - so 12.2-RELEASE will have this fixed.

So - sorry for ignoring this ticket for such a long time, but in the end, things got debugged and fixed (and, unfortunately, there is nothing in OpenVPN to work around this, except use proto udp4 , as you already discovered). Closing this ticket now.