Certificate via MI with OpenSSL 1.1.1 errors with "Unknown Padding Type"

[dpb587]

Hello, I believe there's a bug when using the management interface for private keys with openssl 1.1.1 (observed on both Debian and macOS).

When using openssl/1.1.0t, I have been able to provide certificates and keys via management interface. The NEEDS-CERTIFICATE and RSA_SIGN steps are executed successfully.

When using openssl/1.1.1d with the same credentials, it executes the NEEDS-CERTIFICATE, but it errors with the following before ever calling RSA_SIGN.

Fri Sep 20 11:24:28 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]192.0.2.123:1194
Fri Sep 20 11:24:28 2019 Socket Buffers: R=[131072->131072] S=[131072->131072]
Fri Sep 20 11:24:28 2019 Attempting to establish TCP connection with [AF_INET]192.0.2.123:1194 [nonblock]
Fri Sep 20 11:24:29 2019 TCP connection established with [AF_INET]192.0.2.123:1194
Fri Sep 20 11:24:29 2019 TCP_CLIENT link local: (not bound)
Fri Sep 20 11:24:29 2019 TCP_CLIENT link remote: [AF_INET]192.0.2.123:1194
Fri Sep 20 11:24:29 2019 TLS: Initial packet from [AF_INET]192.0.2.123:1194, sid=25d11069 5432dddb
Fri Sep 20 11:24:29 2019 VERIFY OK: depth=1, C=USA, O=Cloud Foundry, CN=openvpn
Fri Sep 20 11:24:29 2019 VERIFY KU OK
Fri Sep 20 11:24:29 2019 Validating certificate extended key usage
Fri Sep 20 11:24:29 2019 ++ Certificate has EKU (str) TLS Web Server Authentication, expects TLS Web Server Authentication
Fri Sep 20 11:24:29 2019 VERIFY EKU OK
Fri Sep 20 11:24:29 2019 VERIFY OK: depth=0, C=USA, O=Cloud Foundry, CN=server
                         vvvv
Fri Sep 20 11:24:29 2019 OpenSSL: error:04066076:rsa routines:rsa_ossl_private_encrypt:unknown padding type
Fri Sep 20 11:24:29 2019 OpenSSL: error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib
                         ^^^^
Fri Sep 20 11:24:29 2019 TLS_ERROR: BIO read tls_read_plaintext error
Fri Sep 20 11:24:29 2019 TLS Error: TLS object -> incoming plaintext read error
Fri Sep 20 11:24:29 2019 TLS Error: TLS handshake failed
Fri Sep 20 11:24:29 2019 Fatal TLS error (check_tls_errors_co), restarting
Fri Sep 20 11:24:29 2019 SIGUSR1[soft,tls-error] received, process restarting
Fri Sep 20 11:24:29 2019 Restart pause, 5 second(s)

Unfortunately, I am a bit out of my depth, but the following seems most notable to me:

• Why does openvpn's behavior change based on whether the certificate came from configuration vs management interface? That is, where an embedded certificate/key in a profile works, if those same values come from management, it fails.
• Recompiling with some additional logging, I'm able to see that it is caused by an explicit RSAerr when the padding does not match the expected RSA_PKCS1_PADDING (​source).
  • With 1.1.0 it implicitly uses RSA_PKCS1_PADDING via ​this branch.
  • With 1.1.1 it uses RSA_NO_PADDING via ​this branch.
  • It seems like RSA_PKCS1_PSS_PADDING instead of RSA_PKCS1_PADDING may be related to TLS 1.3 support?
  • There's a similar-sounding ​openssl issue about 1.1.1 now always using RSA_NO_PADDING (but the commentary is difficult for me to understand).
  • Possible similar scenario of ​external private key with TLS 1.3?

Any guidance or insight would be appreciated. Thank you for your time.

Reproduction Steps

For the test environment, I built the dependencies with ​build.sh.

For a smaller repro case, I wrote ​some Go code to run a management server from an existing, regular OpenVPN profile which contains an embedded cert/key.

Then, assuming some existing profile, the following commands demonstrate the problem:

profile=some/existing/openvpn.ovpn

# 1.1.0 success
~/tmp/padding-bug/management-tool "${profile}" management &
~/tmp/padding-bug/with-openssl-1.1.0/sbin/openvpn <( ~/tmp/padding-bug/management-tool "${profile}" profile )

# 1.1.1 failure
~/tmp/padding-bug/management-tool "${profile}" management &
~/tmp/padding-bug/with-openssl-1.1.1/sbin/openvpn <( ~/tmp/padding-bug/management-tool "${profile}" profile )
...snip...
Fri Sep 20 11:24:29 2019 OpenSSL: error:04066076:rsa routines:rsa_ossl_private_encrypt:unknown padding type
Fri Sep 20 11:24:29 2019 OpenSSL: error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib
Fri Sep 20 11:24:29 2019 TLS_ERROR: BIO read tls_read_plaintext error
Fri Sep 20 11:24:29 2019 TLS Error: TLS object -> incoming plaintext read error
Fri Sep 20 11:24:29 2019 TLS Error: TLS handshake failed
Fri Sep 20 11:24:29 2019 Fatal TLS error (check_tls_errors_co), restarting

[Selva Nair]

This is from OpenSSL 1.1.1 defaulting to PSS padding (for TLS 1.2 and 1.3) which it does internally and requests a non-padded signature. But we currently have an implicit assumption that RSA signature from management will use PKCS1_RSA_PADDING. This makes it impossible to prompt for signature when 1.1.1 is in use.

There is a patch to fix this (​https://patchwork.openvpn.net/patch/587/) which is under review and requires some more tweaks.

[dpb587]

Thank you for the prompt reply and explanation. It looks like the patch has been around for a while, so I appreciate you reviving the discussion. If there is anything I can do to help the efforts (probably, namely, feedback or testing) please let me know. In the meantime, I'll track this ticket and the patch [whose number is ironically familiar to my handle :)].

[Gert Döring]

587 has been superseded, but a followup patch was eventually merged into master and 2.5-to-be:

commit 8be444df89be09689ba82e58cbc89452a5fcceca
Author: Arne Schwabe <arne@…>
Date: Wed Dec 4 12:08:36 2019 +0100

Add support for OpenSSL TLS 1.3 when using management-external-key

so you could test that... debian packages are also on the build webserver.

[tct]

We have a similar error here:
​https://forums.openvpn.net/viewtopic.php?f=6&t=30880#p94141

[becm]

Switching to OpenSSL 1.1.1 and the new padding for TLS 1.2+ affects 3 distinct operation modes:

When OpenSSL handles key operations internally, everything (kind of) just works™.

The issue linked by tincantech is relate to PKCS#11 hardware tokens, the solution in those cases is to use an up-to-date pkcs11-helper 1.26 (recent OpenVPN 2.5 Windows builds or debian bullseye).
On Windows there is also ​CNG for this use case, new padding is ​supported with OpenVPN 2.5.

The management protocol is extended in OpenVPN 2.5 to support the new padding type (this issue).
The controlling program must also set the version high enough and identify/support the requested padding.