wiki:Concepts-PolicyRouting-Linux

Version 2 (modified by krzee king, 7 years ago) (diff)

--

Introduction

When redirecting Internet-bound traffic across OpenVPN, the local system will be unable to properly route externally-initiated requests back to the originating source. This means that if a service (like ssh or http) is served on the physical uplink, it will cease to work after redirecting traffic.

There are two solutions to the problem. If you know in advance the external source of the connection, you can add a higher-priority route using the local gateway. This is not generally the case, and so will not work as a generic solution.

The second solution is what's called policy routing. This makes a routing decision based on a defined policy that acts on routing rules. On Linux, this setup will require a few steps.

First, connections arriving on the physical uplink are tagged with a connection-specific mark (known as the connmark, or ctmark for “Connection Tracking” mark.) Using connmark is important because it will also correctly route the outbound traffic for related packets, like ICMP error replies.

Second, outbound packets have the ctmark cloned to the per-packet mark (also known as the fwmark.) This is done because routing rules cannot see the ctmark, only the fwmark.

Finally, a routing rule and secondary table are set up. The rule matches the marked traffic (ie: from the physical uplink) and the secondary table routes return packets to the LAN gateway, not via the VPN.

Also note that the kernel's rp_filter feature must be set to 2 (loose mode) or 0 (disabled.) While the kernel default is 0, many distributions set this to 1 (strict mode) in init scripts or sysctl.conf. You must alter this behavior if this applies to your system.

Configuration

Below is an example configuration. You should consult your distro docs for the proper way to do this as there are distro-specific ways to set up routing rules, routing entries, and firewall config. This is a very basic example only that should be integrated to your system as required.

First, the packet tagging. We'll assume your uplink is eth0, so use the following Netfilter rules on the mangle table where appropriate for your setup:

-A PREROUTING -m conntrack --ctstate NEW -i eth0 -j CONNMARK --set-mark 0x1
-A OUTPUT -m connmark --mark 0x1 -j CONNMARK --restore-mark

Next, set up a secondary routing table and the rule to direct traffic here. The number chosen here is arbitrary but must be unique on your system. You may also use a named table by adding one to /etc/iproute2/rt_tables (consult the iproute2 docs for details.) This example uses routing table 100, and assumes your local gateway is 192.168.0.1.

ip route add default via 192.168.0.1 table 100
ip rule add fwmark 0x1 table 100

Be sure to verify the state of rp_filter, and adjust your OS to set this to 2 or 0 as you prefer. Remember that the kernel will use the highest value between the 'all' and interface-specific value, so you can set this to 2 on your uplink (eth0 in this example) interface to over-ride a distro-value of 1.

Conclusion

Redirecting Internet-bound traffic across a VPN has potential uses, but it complicates routing situations when you do actually want some traffic to be sent back out over your uplink. Using the conntrack system under Netfilter on Linux, it is possible to identify traffic that was initially sent via the physical interface and keep it routed back out that direction.

Remember to integrated the above samples properly into your distro startup. The route/rule setup should have a place in your networking configuration files. Most distros also provide a way to load Netfilter rules in iptables-save syntax, which is generally a wrapper around the iptables-restore command. Place the above rule segments into such a file and your distro init will pick it up at the proper time during boot.

Note on another way to do it

When I read the above write-up it taught me another way to achieve a goal that I already knew how to solve.
The problem is when a server that has services listening on the internet, and then runs openvpn with redirect-gateway, the server loses its ability to host its services on its physical interface. Packets get to the service, but the servers response gets routed over the VPN and lost. A clear difference between those return packets and packets generated from the server is the source address. When the IP on the physical device is contacted, it will reply with the IP from the physical device as its source. When the server is generating traffic it will have the source IP of the device that you route through, so the VPN device.
If we assume the servers physical device is 10.0.0.2 and its gateway is 10.0.0.1 then the following commands should solve the problem:

ip route add default via 10.0.0.1 table 10
ip rule add from 10.0.0.2 table 10

The first command adds a second routing table (table 10) with the normal default route (the one the server uses without a vpn)
The second command tells the server to route any packets with src 10.0.0.2 out of the table that we just made (table 10 in our case)