OpenSSL: error:C506D064:microsoft cryptoapi:NCryptSignHash:Ungltige Optionen angegeben

[svnmllr]

Hi,

we got an error message when connecting to our openvpn server:

.\openvpn.exe --config .\config.ovpn --cryptoapicert "THUMB:291C2F01975D895F2C841802E99452E7629C63E5" --verb 11

...
2021-03-10 22:38:39 us=757760 SSL state (connect): TLSv1.3 read server certificate verify
2021-03-10 22:38:39 us=761717 SSL state (connect): SSLv3/TLS read finished
2021-03-10 22:38:39 us=761717 SSL state (connect): SSLv3/TLS write change cipher spec
2021-03-10 22:38:39 us=773724 SSL state (connect): SSLv3/TLS write client certificate
2021-03-10 22:38:39 us=773724 cryptoapicert: enter pkey_rsa_sign_init
2021-03-10 22:38:39 us=777673 cryptoapicert: PSS padding using saltlen = 32
2021-03-10 22:38:39 us=778672 cryptoapicert: calling priv_enc_CNG with alg = SHA256
2021-03-10 22:38:39 us=788676 Signing hash using CNG: data size = 32 padding = 8
2021-03-10 22:38:39 us=789641 SSL alert (write): fatal: internal error
2021-03-10 22:38:39 us=789641 OpenSSL: error:C506D064:microsoft cryptoapi:NCryptSignHash:Ungltige Optionen angegeben
2021-03-10 22:38:39 us=790639 OpenSSL: error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib
2021-03-10 22:38:39 us=793631 TLS_ERROR: BIO read tls_read_plaintext error
...

It seems like the native crypto api throws an "invalid options"-error.
Sadly it is difficult to reproduce because the certificate (& private key) is generated using scep server.
The private key is stored in TPM and not exportable.

We would like to share more details but we do not know which details are relevant.

Best regards,

[Selva Nair]

Probably your TPM cannot generate RSA-PSS signature. TPM 2.0 potentially supports it (TPM 1.2 does not) and PSS is required for TLS 1.3. Even with TLS1.2, OpenSSL 1.1.1+ defaults to PSS.

If lack of RSA-PSS in your TPM is the problem, your only option may be to downgrade to TLS 1.1 (--tls_version_max 1.1 in config) or generate a new cert/key in newer hardware / windows cert store.

[Gert Döring]

Is there anything we can do on the OpenVPN side? It does not look like it, so we should then close the ticket.

(Going through all tickets with "milestone 2.5.1", which has already be missed...)

[Selva Nair]

This is essentially the same as #1296. See the discussion there.

[Gert Döring]

So I'll close this as DUP of #1296.

(I thought it looked familiar but didn't go out searching. Thanks for the pointer)