Opened 18 months ago

Last modified 11 months ago

#952 new Bug / Defect

"comp-lzo no" and "compress" options not compatible

Reported by: ruisantos Owned by:
Priority: major Milestone:
Component: Configuration Version: OpenVPN 2.4.4 (Community Ed)
Severity: Not set (select this one, unless your'e a OpenVPN developer) Keywords:
Cc:

Description

Hello,

I am trying to upgrade our OpenVPN servers from v2.3 to v2.4.

Almost everything is done, with the exception of compression.

Environment:

  • OpenVPN v2.4.4 on server side
  • A mix of OpenVPN 2.3 on the client side, but v2.4.4 behaves the same.

Objective: Upgrade servers from v2.3 to v2.4, maintaining backward compatibility with v2.3 clients and allow v2.4 clients to connect.

As I have mentioned above, the issue is with compression. Currently, we have:

  • Servers v2.3: comp-lzo no
  • Clients: comp-lzo yes

As comp-lzo is a deprecated flag, I was trying to use the compress one to replace it.
According to the docs: If the algorithm parameter is empty, compression will be turned off, but the packet framing for compression will still be enabled, allowing a different setting to be pushed later.
This is mostly true, but communications between the v2.4 server and v2.3 client will not happen with these settings:

  • Server: compress
  • Client: comp-lzo no

After a few debugging, OpenVPN initializes the compression setting with:

  • compress option, with no arguments:
    • comp.alg=1
    • comp.flags=4
  • comp-lzo no option:
    • comp.alg=1
    • comp.flags=0

The COMP_F_SWAP flag in compress option, will swap one byte, which apparently make the protocol incompatible.

Now, if we take a look at other settings:

  • compress lzo option:
    • comp.alg=2
    • comp.flags=0
  • comp-lzo yes option:
    • comp.alg=2
    • comp.flags=0

The source code, however, intentionally defines compress (without options) like that:
options->comp.alg = COMP_ALG_STUB;
options->comp.flags = COMP_F_SWAP;

Bottom line, the issue is that, although there is a way to render lzo compression compatible with v2.3/v2.4 clients, when it is activated, there is no apparently solution for when compression packet framing only is enabled.
IMO, it makes sense to have "compress lzo"/"comp-lzo yes" and "compress"/"comp-lzo no" to be compatible among them.
Unless that COMP_F_SWAP really needs to be there.
Or am I missing something?

Thank you for your help,
Rui

Change History (6)

comment:1 Changed 18 months ago by Gert Döring

It's somewhat of a mess.

I'd suggest to always turn on "comp-lzo yes" on the 2.3 clients, and decide on the 2.4 server what to do in a --client-connect script - a 2.4 client will tell the server what it can do, and --compress is pushable in 2.4, and a per-client setting.

My magic compression handling script (called via --client-connect $script looks roughly like this:

#!/bin/sh
CONF=$1
# compression autoselect
if [ "$IV_LZ4v2" ] ; then
    cat <<EOF >>$CONF
compress lz4-v2
push "compress lz4-v2"
EOF
elif [ "$IV_LZ4" ] ; then
    cat <<EOF >>$CONF
compress lz4
push "compress lz4"
EOF
else
    echo "compress lzo" >>$CONF
fi

exit 0

The IV_<alg> variables are advertised by the client if it can do LZ4v2 (more efficient framing) or LZ4 compression (visible in the server log).

(Caveat: my script is larger due to lots of testing in there, so this is a trimmed-down version which might not be fully correct - but you get the idea)

comment:2 Changed 18 months ago by ruisantos

Thanks a lot for your time in here.

I actually had the same idea to activate lz4 on the v2.4 clients. Good to know someone else thinks like me ;)
Although I knew of the lz4-v2 protocol, I was not aware that it was already production ready. I'll give it a test when the migration is complete.

However, all examples you provide here, imply some kind of compression on the server side. We are thinking of implementing lz4, but in the meantime I would like to keep the same setup: lzo activated on the client side, and deactivated on the server side (stub). We do keep a few thousand connections on our servers, and we've come to realize that, for our workload, it is actually harmful.
Our current setup (v2.3) is:

  • Server: comp-lzo no
  • Client: comp-lzo yes

Is there any way I can mimic this setup on v2.4 server?
Maybe start with comp-lzo no, and then tweak it on ccd!
I will do some tests and will report back the results.

Once again, thank you for your help.

comment:3 Changed 18 months ago by ruisantos

So, after my experiments, I was unable to mimic this behavior.

Option 1)

  • Server: compress lzo
  • Client: comp-lzo yes/no

Option 2)

  • Server: compress or "comp-lzo no"
  • Client: comp-lzo yes

Server: v2.4.4
Client: v2.3.8

Option 1 works, but you are limited to enabling compression on the server side.
Option 2 does not truly work:

  • With "comp-lzo no" on the client, all begins to work
  • With "comp-lzo yes" on the client, larger packets will not be replied. ex: "ping -s 801 8.8.8.8", but with a lower ping size, the icmp reply is received, ex: "ping 8.8.8.8"

So, from what I can see, there is no way to have compression disabled on the server side, without also disabling it on the client side.

Also, when using v2.4.3 as a client, and the following settings are used:

  • Server (v2.4.4): compress
  • Client (v2.4.3): compress lzo

no packet is even replied.
Shouldn't this work also?

Cheers

comment:4 Changed 13 months ago by Clodo

I do some tests with every combination and i confirm there are issues.
I cannot find a configuration server-side (2.4.5) with "compress" and without "comp-lzo"
that can works with a 2.3 client with "comp-lzo no".

With "compress" server-side (without param = no) and "comp-lzo no" client-side (both 2.4.5) throw

Error: Bad compression stud decompression header, disconnect

Note:
with a 2.4.5 server with "compress" and a 2.4.5 client without any related directive, client throw

WARNING: 'comp-lzo' is present in remote config but missing in local config, remote='comp-lzo'

so at least there are messages that need fixes.

comment:5 Changed 11 months ago by ratnix

Found this ticket after spending time discovering the issue on my own.

The issue lies in the 2.4 code, in src/openvpn/compstub.c and its stub_decompress.

If one side is 2.4 and has comp-lzo no in its config, that side ends up using COMP_ALG_STUB.
If the other side uses compression, (comp-lzo yes, compress lzo, or most frustratingly, comp-lzo adaptive and you wait long enough), the other side will send a compressed frame that this side cannot handle if 2.4, but would have accepted and decompressed if 2.3.

In 2.4 the stub code does not know how to decompress lzo, and we end up with broken interop. The compstub.c only exists to support comp-lzo no, which is deprecated and going away, so theoretically you could do anything you want in compstub.c since it only supports a dying option.

There's 3 options that I can see:

  1. Do nothing in code
    • comp-lzo no will just behave differently until it dies
    • too-bad-so-sad for people with this config
    • Bug 861 flirts near the issue but got closed as notabug. IMO that's an error: these issues require a docs fix at minimum.
  2. Simple LZO fix, comp-lzo no gets its 2.3-style behavior back
    • Port most of lzo_decompress from lzo.c into compstub.c (or a cloned file ala comp-lzo-stub.c)
    • 2.4 in this situation can accept LZO frames as it used to, but respects stub's behavior in never sending them
    • This covers the majority case (interoperability with 2.3-and-earlier)
    • Does not play well with lz4, intentionally.
  3. Full compression fix: 'be liberal in what you accept'
    • Port lzo_decompress AND lz4*_decompress into compstub.c
    • option 2 covers lzo, but sets up a case where compress lzo <-> 2.4 comp-lzo no works but lz4 doesn't
    • lz4 arguably shouldn't work with compress lz4 <-> 2.4 comp-lzo no, but 'be liberal in what you accept' might say to do so
    • I don't actually like option 3, but wanted to mention it for completeness.

comment:6 Changed 11 months ago by tincantech

CC

Note: See TracTickets for help on using tickets.