netlink routes does not work with "push route"

[dpalumbo]

Moving to openvpn 2.5, netlink is now used in place of ifconfig/route commands
​https://github.com/OpenVPN/openvpn/blob/release/2.5/Changes.rst

On Linux, if configured without --enable-iproute2, configuring IP >addresses and adding/removing routes is now done via the netlink(3) kernel interface. This is much faster than calling ifconfig or route and also enables OpenVPN to run with less privileges.

What is happening, is that when the interface / routing is created with netlink, this is the routing entry that we see:
172.17.14.0/24 via 172.17.14.1 dev tun-su
while with the old route we have:
172.17.14.0/24 dev tun-su proto kernel scope link src 172.17.14.127

Because of that (subnet not "known" by the kernel) any further command to add a route via push, eg
push "route 172.17.16.0 255.255.255.0 vpn_gateway 100"

Fail with

2021-08-27 10:49:45 us=890376 net_route_v4_add: 172.17.16.0/24 via 172.18.14.1 dev [NULL] table 0 metric 100
2021-08-27 10:49:45 us=890389 sitnl_send: rtnl: generic error (-101): Network is unreachable

This is not necessarily an error in push, because also trying to add the route manually gives this error:

# route add -net 172.17.16.0/24 gw 172.17.14.1 metric 100
SIOCADDRT: Network is unreachable
#

To me, seems that the whole "push" mechanism has been compromised, but i've not performed extensive test.

This bug is also described in
​https://bbs.archlinux.org/viewtopic.php?id=260625
But the workaround is not working for me.

I've replicated the bug in Debian 11.

openvpn server config

[...]
push "topology subnet"

ifconfig 172.17.14.1 255.255.255.0
ifconfig-pool 172.17.14.126 172.17.14.252

route 172.17.14.0 255.255.255.0
push "route 172.17.14.0 255.255.255.0"
push "route-gateway 172.17.14.1"
push "route 172.17.16.0 255.255.255.0 vpn_gateway 100"

mode server
topology subnet
tls-server
[...]

Recompiling the package with --with-iproute2 fix the issue.

[dpalumbo]

I've tracked down with strace the route added by netlink

sendmsg(6, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=32, type=RTM_NEWADDR, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_REPLACE|NLM_F_CREATE, seq=1630050395, pid=
0}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=0, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("tun-su")}, {{nla_len=8, nla_type=IFA_LOCAL}, inet_addr("172.18.14.127")}}, iov_len=32}], msg_iovlen=1, ms
g_controllen=0, msg_flags=0}, 0) = 32

sendmsg(6, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=44, type=RTM_NEWROUTE, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_REPLACE|NLM_F_CREATE, seq=1630050395, pid
=0}, {rtm_family=AF_INET, rtm_dst_len=24, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN, rtm_protocol=RTPROT_BOOT, rtm_scope=RT_SCOPE_UNIVERSE, rtm_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_DST},
 inet_addr("172.18.14.0")}, {{nla_len=8, nla_type=RTA_GATEWAY}, inet_addr("172.18.14.1")}]}, iov_len=44}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 44

And the failing following push route

sendmsg(6, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=52, type=RTM_NEWROUTE, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_REPLACE|NLM_F_CREATE, seq=1630050395, pid
=0}, {rtm_family=AF_INET, rtm_dst_len=32, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN, rtm_protocol=RTPROT_BOOT, rtm_scope=RT_SCOPE_UNIVERSE, rtm_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_DST},
 inet_addr("172.16.66.2")}, {{nla_len=8, nla_type=RTA_GATEWAY}, inet_addr("172.18.14.1")}, {{nla_len=8, nla_type=RTA_PRIORITY}, 100}]}, iov_len=52}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 52
recvmsg(6, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=72, type=NLMSG_ERROR, flags=0, seq=1630050395, pid=18163}, {error=-ENETUNREACH, msg={{len=52, type=R
TM_NEWROUTE, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_REPLACE|NLM_F_CREATE, seq=1630050395, pid=0}, {rtm_family=AF_INET, rtm_dst_len=32, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN, rtm_protocol=RTPROT_BOOT, rtm_sco
pe=RT_SCOPE_UNIVERSE, rtm_type=RTN_UNICAST, rtm_flags=0}, [{{nla_len=8, nla_type=RTA_DST}, inet_addr("172.16.66.2")}, {{nla_len=8, nla_type=RTA_GATEWAY}, inet_addr("172.18.14.1")}, {{nla_len=8, nla_type=RTA_PRIORITY}
, 100}]}}}, iov_len=16384}], msg_iovlen=1, msg_controllen=0, msg_flags=0}, 0) = 72

From
​http://manpages.ubuntu.com/manpages/bionic/man7/rtnetlink.7.html

That should be the difference (to be further analyzed):
RT_SCOPE_UNIVERSE global route
RT_SCOPE_LINK route on this link

RTPROT_KERNEL by the kernel
RTPROT_BOOT during boot

[dpalumbo]

Tackling this issue with gdb, the issue happens after a sitnl_route_add.

The line that was breaking up the configuration was:

push "route 172.18.14.0 255.255.255.0"

This config entry is unneeded (probably needed in an old version, or just my mistake).
Without that line, the proper routing table is populated.

That's anyway IMHO an improvement in the code, as either this should be ignored, or should not modify the existing routing table (which has been just created by openvpn itself).
And/or ignore a push route with the vpn_gateway set in the pushed route.
Finally as netlink report the changed route (it should at least) also that logs could be useful.

This is the log entry when openvpn start with "wrong" push:

2021-08-30 12:05:32 us=646249 net_addr_v4_add: 172.18.14.127/24 dev tun-su
2021-08-30 12:05:32 us=646285 net_route_v4_add: 172.18.14.0/24 via 172.18.14.1 dev [NULL] table 0 metric -1
2021-08-30 12:05:32 us=646303 net_route_v4_add: 172.17.16.0/24 via 172.18.14.1 dev [NULL] table 0 metric 100
[...]

This is the log entry when openvpn start with "correct" push:

2021-08-30 12:13:09 us=353535 net_addr_v4_add: 172.18.14.127/24 dev tun-su
2021-08-30 12:13:09 us=353571 net_route_v4_add: 172.17.16.0/24 via 172.18.14.1 dev [NULL] table 0 metric 100
[...]

See the line:
​https://github.com/OpenVPN/openvpn/blob/5a5d11a0dea51e709b44dfabc1ec97b5f3c5b222/src/openvpn/networking_sitnl.c#L1147

according to this, the gateway is not specified in the caller (but specified afterward via push "route-gateway 172.18.14.1"), which have the route modified by netlink.
likely, this was not happening on the previous version because route command was not smart enough to modify the existing entry.

HTH

[tct]

Can you try changing this in your server config:

push "topology subnet"

ifconfig 172.17.14.1 255.255.255.0
ifconfig-pool 172.17.14.126 172.17.14.252

To:

push "topology subnet"

server 172.17.14.0 255.255.255.0 nopool
ifconfig-pool 172.17.14.126 172.17.14.252

[dpalumbo]

Hi @tct,

I'm not sure why you asked that, but i believe that --server would do much more.

According to the manual, this is expanded as:

                   mode server
                   tls-server
                   push "topology [topology]"

                   if dev tun AND (topology == net30 OR topology == p2p):
                     ifconfig 10.8.0.1 10.8.0.2
                     if !nopool:
                       ifconfig-pool 10.8.0.4 10.8.0.251
                     route 10.8.0.0 255.255.255.0
                     if client-to-client:
                       push "route 10.8.0.0 255.255.255.0"
                     else if topology == net30:
                       push "route 10.8.0.1"

                   if dev tap OR (dev tun AND topology == subnet):
                     ifconfig 10.8.0.1 255.255.255.0
                     if !nopool:
                       ifconfig-pool 10.8.0.2 10.8.0.253 255.255.255.0
                     push "route-gateway 10.8.0.1"
                     if route-gateway unset:
                       route-gateway 10.8.0.2

Which is not what i'm looking for, and make no sense to me.

Anyhow, as reported, the fix is to remove

push "route 172.18.14.0 255.255.255.0"

Because, with netlink, this causes the route to be re-written.
This route is already pushed by openvpn as default, as this is the subnet pushed by ifconfig.

[tct]

Replying to dpalumbo:

Tackling this issue with gdb, the issue happens after a sitnl_route_add.

The line that was breaking up the configuration was:

push "route 172.18.14.0 255.255.255.0"

This config entry is unneeded (probably needed in an old version, or just my mistake).
Without that line, the proper routing table is populated.

Replying to dpalumbo:

as reported, the fix is to remove

push "route 172.18.14.0 255.255.255.0"

Can we close this now ?

[dpalumbo]

I believe that the correct answer is that OpenVPN should intercept the wrong route and ignore it.
Do you need a dedicated ticket?
This one seems to me that contains the proper information.

[Gert Döring]

@ordex, I have only skimmed this - but this is your field. Can we fix this? (As in: do what iproute2 does, which might be "ignore exact match routes for on-link ifconfig networks")

This IPv4 stuff...

[tct]

Replying to dpalumbo:

What is happening, is that when the interface / routing is created with netlink, this is the routing entry that we see:
172.17.14.0/24 via 172.17.14.1 dev tun-su
while with the old route we have:
172.17.14.0/24 dev tun-su proto kernel scope link src 172.17.14.127

Because of that (subnet not "known" by the kernel) any further command to add a route via push, eg
push "route 172.17.16.0 255.255.255.0 vpn_gateway 100"

Fail with

2021-08-27 10:49:45 us=890376 net_route_v4_add: 172.17.16.0/24 via 172.18.14.1 dev [NULL] table 0 metric 100
2021-08-27 10:49:45 us=890389 sitnl_send: rtnl: generic error (-101): Network is unreachable

This is not necessarily an error in push, because also trying to add the route manually gives this error:

# route add -net 172.17.16.0/24 gw 172.17.14.1 metric 100
SIOCADDRT: Network is unreachable
#

Note:

net_route_v4_add: 172.17.16.0/24 via 172.18.14.1

Where is 172.18.14.1 ?

[dpalumbo]

Where is 172.18.14.1 ?

172.18.14.1 is the same server as 172.17.14.1, but running TCP (172.18) in place of UDP (172.17).
I was using it for several tests, so please consider this IP to be as 172.17.14.1.
In short,

• when ifconfig is 172.17.14.1, the error is on the "push route 172.17.14.0 [...]"
• when ifconfig is 172.18.14.1, the error is on the "push route 172.18.14.0 [...]"

do what iproute2 does, which might be "ignore exact match routes for on-link ifconfig networks

Catch exactly the point.
Please note, iproute2 is not ignoring it, it does have an error when the relative command is executed (as already reported):

2021-08-27 10:49:45 us=890376 net_route_v4_add: 172.17.16.0/24 via 172.18.14.1 dev [NULL] table 0 metric 100
2021-08-27 10:49:45 us=890389 sitnl_send: rtnl: generic error (-101): Network is unreachable

It could be intercepted by OpenVPN even before the netlink code, eventually.

HTH

[Antonio Quartulli]

Thanks for the report. I think this is the same issue reported in #1378 (add a route to the same subnet already configured via ifconfig).

This is obviously a configuration error, but we want sitnl to detect the problematic route and just ignore it.

[dpalumbo]

If I'm not wrong, #1378 declare that the route is not added.
Which is the opposite of this issue.

The sitnl code refuses to add routes for targets that are already there as connected route, and I think that this is the correct thing to do (actually, the kernel code refuses this) - but we need a proper error message. Antonio was already volunteered to fix this.

In this case sitnl is not refusing to do it.

Anyhow, the final goal (ignore a wrongly pushed route) is definitely going to fix the issue.

[Antonio Quartulli]

For this specific issue, we would need to check if the route that we want to add is overlapping with the subnet configured by ifconfig. if so, we should just skip the route and move on.