Opened 8 years ago

Last modified 15 months ago

#626 accepted Bug / Defect

Routing entry not deleted when MAC address moves from client to server

Reported by: zewaren Owned by: Gert Döring
Priority: minor Milestone: release 2.7
Component: Networking Version: OpenVPN git master branch (Community Ed)
Severity: Not set (select this one, unless your'e a OpenVPN developer) Keywords:
Cc:

Description

Context:

A TAP OpenVPN connection is made between two virtual machine hypervisors. The TAP devices are bridged with the VM interfaces, thus creating a L2 bridge between the VMs on different hosts.

A virtual machine (192.168.42.102) is moved from a hypervisor host (node02) to another one (node01).

Initial situation:

.---------------------------------.                         .---------------------------------.
| node01                          |                         | node02                          |
| OpenVPN server                  |                         | OpenVPN client                  |
|                                 |                         |                                 |
|    .-----------------------.    |                         |     .-----------------------.   |
|    | br0                   |    |                         |     | br0                   |   |
|    | .-------------------. |    |                         |     | .-------------------. |   |
|    | | tap0              | |    |                         |     | | tap1              | |   |
|    | | 192.168.42.1      |<----------------OpenVPN----------------| 192.168.42.2      | |   |
|    | | de:57:b6:64:0e:c8 | |    |                         |     | | 2e:60:ce:c9:63:d1 | |   |
|    | '-------------------' |    |                         |     | '-------------------' |   |
|    |                       |    |                         |     | .-------------------. |   |
|    |                       |    |                         |     | | vnet0             | |   |
|    |                       |    |                         |     | | 192.168.42.102    | |   |
|    |                       |    |                         |     | | 52:54:00:de:ad:02 | |   |
|    |                       |    |                         |     | '-------------------' |   |
|    '-----------------------'    |                         |     '-----------------------'   |
'---------------------------------'                         '---------------------------------'

Final situation:

.---------------------------------.                         .---------------------------------.
| node01                          |                         | node02                          |
| OpenVPN server                  |                         | OpenVPN client                  |
|                                 |                         |                                 |
|    .-----------------------.    |                         |     .-----------------------.   |
|    | br0                   |    |                         |     | br0                   |   |
|    | .-------------------. |    |                         |     | .-------------------. |   |
|    | | tap0              | |    |                         |     | | tap1              | |   |
|    | | 192.168.42.1      |<----------------OpenVPN----------------| 192.168.42.2      | |   |
|    | | de:57:b6:64:0e:c8 | |    |                         |     | | 2e:60:ce:c9:63:d1 | |   |
|    | '-------------------' |    |                         |     | '-------------------' |   |
|    | .-------------------. |    |                         |     |                       |   |
|    | | vnet0             | |    |                         |     |                       |   |
|    | | 192.168.42.102    | |    |                         |     |                       |   |
|    | | 52:54:00:de:ad:02 | |    |                         |     |                       |   |
|    | '-------------------' |    |                         |     |                       |   |
|    '-----------------------'    |                         |     '-----------------------'   |
'---------------------------------'                         '---------------------------------'

What I do:
Ping 192.168.42.2 (node02) from 192.168.42.102 (the VM).

What happens:
Once the machine is moved from node02 to node01, no ping reply reaches the VM.

What should happen instead:
Ping replies should reach the VM.


OpenVPN version:

OpenVPN 2.3.8 x86_64-redhat-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Aug  4 2015
library versions: OpenSSL 1.0.1e-fips 11 Feb 2013, LZO 2.06
Originally developed by James Yonan
Copyright (C) 2002-2010 OpenVPN Technologies, Inc. <sales@openvpn.net>
Compile time defines: enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dependency_tracking=no enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_http_proxy=yes enable_iproute2=yes enable_libtool_lock=yes enable_lzo=yes enable_lzo_stub=no enable_management=yes enable_multi=yes enable_multihome=yes enable_pam_dlopen=no enable_password_save=yes enable_pedantic=no enable_pf=yes enable_pkcs11=yes enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_pthread=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_socks=yes enable_ssl=yes enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=yes enable_win32_dll=yes enable_x509_alt_username=yes with_crypto_library=openssl with_gnu_ld=yes with_iproute_path=/sbin/ip with_mem_check=no with_plugindir='$(libdir)/openvpn/plugins' with_sysroot=no

Linux version:

CentOS Linux release 7.1.1503 (Core)
Linux node02 4.1.12-default #1 SMP Thu Nov 5 16:00:51 CET 2015 x86_64 x86_64 x86_64 GNU/Linux

Server config:

local 198.51.100.7
port 1194
proto udp
dev tap0
mode server
tls-server
ca /etc/openvpn/pki/whatever-ca-crt.pem
cert /etc/openvpn/pki/whatever-server-crtchain.pem
key /etc/openvpn/pki/whatever-server-key.pem
dh /etc/openvpn/pki/whatever-ca-dh.pem
tls-auth /etc/openvpn/pki/ta.key 0
server-bridge 192.168.11.1 255.255.255.0 192.168.11.100 192.168.11.200
client-to-client
duplicate-cn
keepalive 10 120
#comp-lzo
max-clients 4
user nobody
group nobody
persist-key
persist-tun
status openvpn-status.log
log-append openvpn.log
verb 9
script-security 2

Client config:

client
dev tap1
proto udp
remote 198.51.100.7 1194
resolv-retry infinite
nobind
persist-key
persist-tun
ca /etc/openvpn/pki/whatever-ca-crt.pem
cert /etc/openvpn/pki/whatever-client02-crtchain.pem
key /etc/openvpn/pki/whatever-client02-key.pem
ns-cert-type server
tls-auth /etc/openvpn/pki/ta.key 1
#comp-lzo
verb 9
log-append openvpnclient.log
script-security 2
keepalive 30 120
float
status openvpn-status.log

Here are the log excerpt for a ping packet from the VM the other hypervisor:

Server sends ping packet:

read from TUN/TAP returned 98
GET INST BY VIRT: 2e:60:ce:c9:63:d1 -> OpenVPNClient02/198.51.100.8:45186 via 2e:60:ce:c9:63:d1
OpenVPNClient02/198.51.100.8:45186 TUN READ [98]
OpenVPNClient02/198.51.100.8:45186 ENCRYPT FROM: 0000006c 2e60cec9 63d15254 00dead02 08004500 0054f4ea 0000ff01 f104c0a[more...]
OpenVPNClient02/198.51.100.8:45186 ENCRYPT TO: 8bc767c2 ebacf0f8 120e8996 092c2d68 675bd569 33189ea8 2aa88961 998fcfb[more...]
OpenVPNClient02/198.51.100.8:45186 UDPv4 WRITE [133] to [AF_INET]198.51.100.8:45186: P_DATA_V1 kid=0 DATA c2fe31ee 55bf2043 c24309d1 fef442c3 78d1002d 8bc767c2 ebacf0f8 120e899[more...]
OpenVPNClient02/198.51.100.8:45186 UDPv4 write returned 133

Client receives ping packet:

UDPv4 READ [133] from [AF_INET]198.51.100.7:1194: P_DATA_V1 kid=0 DATA c2fe31ee 55bf2043 c24309d1 fef442c3 78d1002d 8bc767c2 ebacf0f8 120e899[more...]
DECRYPT TO: 0000006c 2e60cec9 63d15254 00dead02 08004500 0054f4ea 0000ff01 f104c0a[more...]
PID_TEST [0] [SSL-0] [112233445566778899>>>>>>>>>>>>>>>>EEEEEEEEEEEEEEEEEEEEEEEEEEEEEE] 0:107 0:108 t=1447061166[0] r=[-2,64,15,0,1] sl=[21,64,64,528]
TUN WRITE [98]
write to TUN/TAP returned 98

Client sends pong packet:

TUN READ [98]
ENCRYPT FROM: 0000006a 525400de ad022e60 cec963d1 08004500 00541a02 00004001 8aeec0a[more...]
ENCRYPT TO: 7cc560fa 28a6349d bad24eca 5fee496c b570067a 0b0e3d6e 4f5f518f 4615351[more...]
UDPv4 WRITE [133] to [AF_INET]198.51.100.7:1194: P_DATA_V1 kid=0 DATA 31716baa 29bd16a4 68e4a888 a2ffccca 673cbd94 7cc560fa 28a6349d bad24ec[more...]
UDPv4 write returned 133

Server receives pong packet and sends it back to client:

OpenVPNClient02/198.51.100.8:45186 UDPv4 READ [133] from [AF_INET]198.51.100.8:45186: P_DATA_V1 kid=0 DATA 31716baa 29bd16a4 68e4a888 a2ffccca 673cbd94 7cc560fa 28a6349d bad24ec[more...]
OpenVPNClient02/198.51.100.8:45186 DECRYPT TO: 0000006a 525400de ad022e60 cec963d1 08004500 00541a02 00004001 8aeec0a[more...]
OpenVPNClient02/198.51.100.8:45186 GET INST BY VIRT: 52:54:00:de:ad:02 -> OpenVPNClient02/198.51.100.8:45186 via 52:54:00:de:ad:02
OpenVPNClient02/198.51.100.8:45186 ENCRYPT FROM: 0000006d 525400de ad022e60 cec963d1 08004500 00541a02 00004001 8aeec0a[more...]
OpenVPNClient02/198.51.100.8:45186 ENCRYPT TO: 5526778c dc9e2db5 64674da6 9d1eab5c c8511d8e bf3ed564 75acdc5c 4dbfa2e[more...]
OpenVPNClient02/198.51.100.8:45186 UDPv4 WRITE [133] to [AF_INET]198.51.100.8:45186: P_DATA_V1 kid=0 DATA b70e40a4 2c9e44c2 43545727 7513c0f7 a61148c1 5526778c dc9e2db5 64674da[more...]
OpenVPNClient02/198.51.100.8:45186 UDPv4 write returned 133

Client receives pong packet:

UDPv4 read returned 133
UDPv4 READ [133] from [AF_INET]198.51.100.7:1194: P_DATA_V1 kid=0 DATA b70e40a4 2c9e44c2 43545727 7513c0f7 a61148c1 5526778c dc9e2db5 64674da[more...]
DECRYPT TO: 0000006d 525400de ad022e60 cec963d1 08004500 00541a02 00004001 8aeec0a[more...]
TUN WRITE [98]
write to TUN/TAP returned 98

TCPdump on client shows the pong packet coming back:

10:38:31.163434 IP 192.168.42.102 > 192.168.42.2: ICMP echo request, id 48223, seq 806, length 64
10:38:31.163472 IP 192.168.42.2 > 192.168.42.102: ICMP echo reply, id 48223, seq 806, length 64
10:38:31.165248 IP 192.168.42.2 > 192.168.42.102: ICMP echo reply, id 48223, seq 806, length 64

Getting the server's status report indicates that the MAC address of 192.168.42.102 (the VM) is still considered to be reachable through the VPN client while it now should be considered a local address:

OpenVPN CLIENT LIST
Updated,Mon Nov  9 10:30:52 2015
Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since
OpenVPNClient02,198.51.100.8:45186,63009,96567,Mon Nov  9 10:21:58 2015
ROUTING TABLE
Virtual Address,Common Name,Real Address,Last Ref
2e:60:ce:c9:63:d1,OpenVPNClient02,198.51.100.8:45186,Mon Nov  9 10:30:51 2015
52:54:00:de:ad:02,OpenVPNClient02,198.51.100.8:45186,Mon Nov  9 10:30:51 2015
GLOBAL STATS
Max bcast/mcast queue length,1
END

Reading the source code indicates that multi_learn_addr() in multi.c does not update routes when the destination addr is local:
https://github.com/OpenVPN/openvpn/blob/ea66a2b5cdb21422139c421b4d3733e1c1c3937e/src/openvpn/multi.c#L1016
Thus, the bad route is kept and is used while it should have been deleted.

When I move the VM between two hypervisors that are both clients of the OpenVPN server (instead of moving it from a client to the server), the condition is respected, the routing table is updated and the network has no problem:

OpenVPNClient02/198.51.100.8:56038 MULTI: Learn: 52:54:00:de:ad:02 -> OpenVPNClient02/198.51.100.8:56038

So, to sum up the diagnostic:
When a MAC address moves from a client to the server, the entry in the routing table is not deleted, and the trafic to the MAC address is sent to the client connection where it was last seen while it should be handled locally.

Should I write and submit a patch?

Attachments (1)

patch-multi_process_incoming_tun.patch (1.6 KB) - added by zewaren 8 years ago.

Download all attachments as: .zip

Change History (7)

comment:1 Changed 8 years ago by Gert Döring

the bane of l2 mac learning...

Yes, a patch that deletes the MAC address from the routing table (= defaulting to 'tap') when it is seen on the local tap side would be useful, and should be safe enough. A log entry ("MULTI: Learn: $mac -> local tap interface" or something) would be good to understand what is happening, especially if addresses are flip-flop'ing

thanks

Changed 8 years ago by zewaren

comment:2 Changed 8 years ago by zewaren

I've attached a patch.

Since it's my first time digging into the code of OpenVPN, I strongly advice core developers to review it.

comment:3 Changed 8 years ago by Gert Döring

Owner: set to Gert Döring
Status: newaccepted

thanks for the patch. Times have been a bit busy, but it is not lost.

comment:4 Changed 5 years ago by takken3

spam

Last edited 5 years ago by Eric Crist (previous) (diff)

comment:5 Changed 5 years ago by tct

cc

comment:6 Changed 15 months ago by Gert Döring

Milestone: release 2.7

*sigh*. I was vaguely aware that we have something here, but was too busy with other aspects. Bumping the milestone to 2.7, need to come back to this.

Note: See TracTickets for help on using tickets.