regressions in openvpn 2.4.1

[lfarkas]

hi,
after we upgrade our servers and client to 2.4.1 we detect many regressions.

• first was that with this the server no longer works and the server restart fail after upgrade. imho it's not a safe behavior. but it was easy to fix at least.

script-security 2 system 

• then the new systemd unit files (ie openvpn-server and openvpn-client) not working. ie if i move all th config file from /etc/openvpn to /etc/openvpn/server then the server fail to start. and still not found any other solution then move back the config files. i open a bugzilla:

​https://bugzilla.redhat.com/show_bug.cgi?id=1446795

• but the most annoying on is that if the server runs and a client already connected but reboot the client then in most case it's not able to reconnect. on the server log we see this error message:

Sun May  7 23:46:57 2017 .. PUSH: client wants to negotiate cipher (NCP), but server has already generated data channel keys, ignoring client request
Sun May  7 23:46:57 2017 ... AEAD Decrypt error: cipher final failed
Sun May  7 23:47:02 2017 ... AEAD Decrypt error: cipher final failed

but if i restart the server then everything working perfectly and a the clients can reconnect.
relevant part of the server config:

proto udp
dev-type tun
dev vpn-udp

remote-cert-tls client
cipher		AES-256-CBC
auth		SHA256
tls-cipher	TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256:TLS-DHE-RSA-WITH-AES-256-CBC-SHA

topology subnet
client-to-client
comp-lzo no
persist-tun
persist-key
persist-local-ip
keepalive 10 120
push "comp-lzo no"
push "persist-tun"
push "persist-key"

[David Sommerseth]

Regarding the script-security 2 system ... That has not been valid since v2.3.0 (released Jan 2013). In v2.3.x the system part was just ignored. As of v2.4.0 the option parser have become much stricter and will complain and halt execution on configuration errors. This is exactly to avoid misleading users they have a certain configuration while it is not doing what they think it was doing.

In regards to the systemd unit files, that is something which is on my plate. I have still not yet been able to reproduce this. I am using both server and client profiles on 3 different EL7 installations of mine, all which are in production. I don't see this issue at all, so I don't fully understand why you see that and I am not able to trigger this.

For the NCP issue, syzzer is the one who knows most about these issues. It would be good to see the client configuration file as well.

[tct]

CC

[Gert Döring]

I would guess that the client is not using nobind, so the server is not recognizing that it's actually a new connection, instead assuming it's the same session again (and then failing to send NCP ciphers).

What I'm wondering a bit is why keepalive on the server isn't catching this - it should declare the client dead after 120 seconds, resetting per-client state, after which a reconnect should succeed.

Anyway, as a quick workaround, just configure disable-ncp on the server.

[lfarkas]

so the most serious is the ncp. it's strange for me since tls-cipher list is also a negotioation list and seems to work perfectly ncp don't. the root of a problem is that we've a few openwrt based routers. the latest released openwrt's is still 2.3.6 which is not support ncp. but we still need to support them. so here comes problems.
so only the

cipher		AES-256-CBC

setting is working since AES-256-GCM not supported on openwrt and therefore de default

ncp-ciphers	AES-256-GCM:AES-128-GCM

also not working!!! so imho it'd be wise to change the default to

ncp-ciphers	AES-256-GCM:AES-128-GCM:AES-256-CBC

after i set it on the server it's seems to working. and yes i don't use the nobind option until now so i set it too (which is in the new default systemd unit file which is good).

any the new systemd unit files are working on the clients and the problem only has the new unit files on the server.

so now with this settings on the server:

remote-cert-tls client
#tls-version-min 1.2
auth		SHA256
cipher		AES-256-CBC
ncp-ciphers	AES-256-GCM:AES-128-GCM:AES-256-CBC
tls-cipher	TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-ECDHE-ECDSA-WITH-AES-256-CBC-SHA:TLS-ECDHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA

tls-version-min also have to commented out because of the openwrt clients.

and on the client:

nobind
client
remote-cert-tls	server
auth		SHA256
cipher		AES-256-CBC

it seems to working but lets wait a few days since the problem happened randomly before.

[lfarkas]

for the systemd server unit file it's still not working. if i copy the same files under the server directory i always got this error:

91.104.120.246:40884 WARNING: Failed to stat CRL file, not (re)loading CRL.
91.104.120.246:40884 WARNING: Failed running command (--tls-verify script): could not execute external program
91.104.120.246:40884 VERIFY SCRIPT ERROR: depth=1, C=HU, L=Budapest, O=Lenux Bt., CN=Lenux Root CA, emailAddress=help@lenux.hu
91.104.120.246:40884 OpenSSL: error:140890B2:SSL routines:SSL3_GET_CLIENT_CERTIFICATE:no certificate returned
91.104.120.246:40884 TLS_ERROR: BIO read tls_read_plaintext error
91.104.120.246:40884 TLS Error: TLS object -> incoming plaintext read error
91.104.120.246:40884 TLS Error: TLS handshake failed
91.104.120.246:40884 SIGUSR1[soft,tls-error] received, client-instance restarting

and it has nothing to do with crl or verify scripts itself since it's there readable etc (it's easy to check that i've got different error message if i rename these files.

[plaisthos]

Your comment about ncp-cipher confuses me. ncp is *only* enabled if both client and server support AES-256-GCM. So please add a log where you show that ncp breaks your 2.3.6 clients (which don't even have NCP)

[plaisthos]

Also hardcoding tls-cipher list is not recommended, you should stick to specifying classes of ciphers, e.g. tls-cipher "DEFAULT:!EXP:!PSK:!SRP:!kRSA". In your cipher list, the majority of the ciphers require a fairly modern SSL library. Your client library not might not have any of these.

[lfarkas]

the reason i've to use

cipher		AES-256-CBC

instead of

cipher		AES-256-GCM

are the old clients. and even if the clients are new this is a better default

ncp-ciphers	AES-256-GCM:AES-128-GCM:AES-256-CBC

then the current default

ncp-ciphers	AES-256-GCM:AES-128-GCM

since it's required GCM client support.

anyway the above tls-cipher list is working on all of the openwrt, and other desktop and server linux distros and android and ios clients too. so it doesn't seems to restrictive. but yes currently openvpn command line are not really usable since

--show-ciphers
--show-digests
--show-engines

shows only those which are supported, BUT

--show-tls

shows even those tls cipher which is not support! eg. on openwrt it list

TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384

but it's clearly not supported on openwrt's version of openvpn.

[plaisthos]

ECHDE cipher support needs OpenVPN 2.4. (See commits if you want more info)

Your proposed ncp-cipher line would only help for clients that have "cipher AES-256-CBC" in their config when the server has something different in the cipher command. A configuration that would not work at all with a pre 2.4 server.

And apart from that NCP is *only* enabled if the SSL library supports GCM.

[David Sommerseth]

 * then the new systemd unit files (ie openvpn-server and openvpn-client) not working. ie if i move all th config file from /etc/openvpn to /etc/openvpn/server then the server fail to start. and still not found any other solution then move back the config files. i open a bugzilla: 

​https://bugzilla.redhat.com/show_bug.cgi?id=1446795

for the systemd server unit file it's still not working. if i copy the same files under the server directory i always got this error:

[....log.data....]

and it has nothing to do with crl or verify scripts itself since it's there readable etc (it's easy to check that i've got different error message if i rename these files.

This I believe is completely bogus, as I've pointed out in the bugzilla. Your configuration uses --user openvpn --group openvpn while the default permission of the directory you have placed your --tls-verify script in is owned by root:root and have 0750 (rwxr-x---). This will not work.

Quick fix: # chmod o+x /etc/openvpn/server

[lfarkas]

agree it's a packaging bug on rhel/fedora

[Steffan Karger]

Replying to lfarkas:

the reason i've to use

cipher		AES-256-CBC

instead of

cipher		AES-256-GCM

are the old clients. and even if the clients are new this is a better default

ncp-ciphers	AES-256-GCM:AES-128-GCM:AES-256-CBC

then the current default

ncp-ciphers	AES-256-GCM:AES-128-GCM

since it's required GCM client support.

That's not how it works. If you need to support non-NCP clients, you either have to ensure the cipher options at both ends matches, or you have to add the cipher to the ncp-ciphers list. So if you have cipher AES-256-CBC, there is no reason to add :AES-256-CBC to ncp-ciphers.

Also, AES-256-CBC has never been a default cipher in OpenVPN, so it makes no sense for us to add it to the default list. If you used cipher AES-252-CBC before, you should simple keep using that in all your configs.

anyway the above tls-cipher list is working on all of the openwrt, and other desktop and server linux distros and android and ios clients too. so it doesn't seems to restrictive. but yes currently openvpn command line are not really usable since

--show-ciphers
--show-digests
--show-engines

shows only those which are supported, BUT

--show-tls

shows even those tls cipher which is not support! eg. on openwrt it list

TLS-ECDHE-ECDSA-WITH-AES-256-GCM-SHA384

but it's clearly not supported on openwrt's version of openvpn.

We improved a bit on this in OpenVPN 2.4, but OpenSSL (and mbed TLS) make it very hard to provide an accurate list. We simply show what the crypto library claims it supports... See #304 for more info.

[Steffan Karger]

Summarizing:

• the system method has been removed a long time ago (2.3.0), and we now just notify you of that, instead of plainly ignoring the option parameter
• the systemd / right issues you have are either rhel/fedora packaging problems, or sysadmin errors, and need not be dicussed here
• the unreliable NCP problems are better described in #887, let's focus the discussion over there.

So I'm closing this ticket.

Also, next time please report one problem per ticket. That makes it a lot easier to keep track of everything.

[Gert Döring]

Replying to plaisthos:

Your comment about ncp-cipher confuses me. ncp is *only* enabled if both client and server support AES-256-GCM. So please add a log where you show that ncp breaks your 2.3.6 clients (which don't even have NCP)

While this ticket is already closed, this particular comment needs more explanation.

"Proper" NCP only happens between 2.4 clients and 2.4 servers - "proper" being that the server can pick a cipher from a list and push it back, and the client will then change what it does to the pushed cipher.

To ease migration away from BF-CBC, 2.4 also supports "poor man's NCP", in which case the server will change the server end to use the cipher that is configured on the client side. So you can have a 2.3 client with cipher AES-256-CBC (no GCM in 2.3...) and via OCC, the client tells the server that "hey, I have AES-256-CBC in my config!". If the server allows NCP, and that cipher is in the --ncp-ciphers list on the server, the server will then move this client context over to AES-256-CBC.

Of course this all blows up if the client has a cipher in his config that is not allowed on the server side... so I can see that this can cause friction.

OTOH I would not extend the default --ncp-ciphers list, because this sort of setup needs a conscious decision by the admin on what to use on the client side, and then he can just add it to the server list. We might need to document it better - for which this ticket is a starting point.