mssfix code is slow

[ValdikSS]

Code that do mssfix (mss_fixup_ipv4 in mss.c) is rather slow. It usually slows down packet rate by half.

cipher none
auth none
tun-mtu 6000
txqueuelen 1000
sndbuf 0
rcvbuf 0

mssfix 0

valdikss@valaptop ~ % iperf -c 192.168.210.1
------------------------------------------------------------
Client connecting to 192.168.210.1, TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  3] local 192.168.210.2 port 60597 connected with 192.168.210.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.06 GBytes   912 Mbits/sec

cipher none
auth none
tun-mtu 6000
txqueuelen 1000
sndbuf 0
rcvbuf 0

mssfix 1450

(mssfix 1450 is the default value. mssfix is enabled by default even if not present in configuration file)

valdikss@valaptop ~ % iperf -c 192.168.210.1
------------------------------------------------------------
Client connecting to 192.168.210.1, TCP port 5001
TCP window size:  256 KByte (default)
------------------------------------------------------------
[  3] local 192.168.210.2 port 60490 connected with 192.168.210.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   602 MBytes   504 Mbits/sec

[Gert Döring]

I find this hard to believe, given that the amount of stuff it does for non-TCP-SYN packets is just a handfull of instructions, as compared to all the crypto, compression and data copying that is going on elsewhere (mss.c, mss_fixup_ipv4()) - even without crypto and compression, it should still be neglible compared to "send the packet from kernel to userland and back"...

My guess is more that "mssfix 1450" is causing UDP packet fragmentation, as the resulting OpenVPN packets will be bigger than 1500 bytes - and *that* will hurt a lot.

Not sure exactly how "mssfix 0" packets look like on the wire if you feed it with a full TCP packet, though - should fragment as well...

Can you check:

• whether you see fragmentation on the outside UDP packets, and if yes, for which case?
• how big the TCP segments *in* the tunnel are in both cases?
• how the SYN looks like on the receiving end (after mssfix code)?

What you could do as well is profile the code (gcc -p/-gp), to see how much time is actually spent in mss_fixup_ipv4()...

[ValdikSS]

mssfix does't just set MSS to the exact value you configure, instead it calculates overhead for specified MTU. mssfix 1450 sets MSS to prevent fragmentation on links with MTU 1450 or higher. I was testing it with direct connection with MTU 1500 so there should be any fragmentation with mssfix 1450, but it should be with mssfix 0. And the bandwidth is higher even with the fragmentation as could be seen in the first post.

Anyway, right now I'm unable to reproduce this on my new PC. I'll set up my old PC which was used in said tests and test it again.

[ValdikSS]

I get it now. My old PC isn't that fast to process small packets at gigabit speed. If we increase tun-mtu up to 6000 and set mssfix to 0, we have huge 6000 byte packets and everything is fast, but if we don't explicitly set mssfix to any value, it it set to 1450 and we never get big packets even if we have tun-mtu 6000.

Everything works with tun-mtu 6000 mssfix 5800 as fast as with mssfix 0.

Sorry for misleading bug.

[Gert Döring]

Thanks for taking the time and re-testing - so in the end, it's similar to my "fragmentation" theory, just on a higher layer - "more packets, more context switching, more overhead, ..."

Now I just wonder if we should do anything, or see this as "documentation people can find" and close the ticket? ("Normal Internet users" won't be able to utilize anything bigger than 1500 byte external MTU, so the fact that mssfix gets in the way for tun-mtu 6000 should rarely be a problem)

[Gert Döring]

Discussed this at the IRC meeting - everyone points at me ("networking -> cron2"), so I decide that we leave it at this. If someone searches trac for mssfix, this ticket is found...

Thanks, valdikss for discussing this, even if it turned out to be not a bug :-)