This document provides a technical overview of the cryptographic layer of OpenVPN, assuming a prior understanding of modern cryptographic concepts. For more details on security aspects related to OpenVPN, refer to [this FAQ item](wiki/295-are-there-any-known-security-vulnerabilities-with-openvpn).

OpenVPN supports two modes of authentication:

- **Static Key**: Utilizes a pre-shared static key.

- **TLS**: Employs SSL/TLS along with certificates for authentication and key exchange.

In this mode, a pre-shared key is created and distributed between the OpenVPN peers before establishing the tunnel. The static key comprises four independent keys:

- HMAC send

- HMAC receive

- Encrypt

- Decrypt

By default, the same HMAC key and encryption/decryption key are used by both hosts in static key mode. However, the `--secret` directive with the direction parameter allows utilizing all four keys independently.

An SSL session with bidirectional authentication is initiated, requiring each connection side to present a valid certificate. Upon successful SSL/TLS authentication, key materials for encryption/decryption and HMAC are randomly generated using OpenSSL's `RAND_bytes` function and exchanged over the SSL/TLS connection. Each connection side contributes random material, ensuring unique keys for send HMAC, receive HMAC, packet encryption, and packet decryption. Depending on the `--key-method` used (1 or 2), keys are derived either directly from the `RAND_bytes` function or using the TLS PRF function. Starting from OpenVPN 1.5.0, `--key-method 2` is available and will become the default in OpenVPN 2.0.

SSL/TLS rekeying includes a `transition-window` parameter allowing key overlap during renegotiations, avoiding latency issues.

To operate over a reliable transport, OpenVPN adds a reliable transport layer atop UDP, as shown in the diagram below.

Once each peer has its keys, tunnel forwarding begins. The packet structure is as follows:

The plaintext inside the encrypted envelope is structured as:

The HMAC and explicit IV are not included within the encrypted envelope.

The per-packet IV is randomized using a nonce-based PRNG, initially seeded by the OpenSSL `RAND_bytes` function.

OpenSSL's EVP interface provides HMAC, encryption, and decryption functions, allowing selection of any cipher, key size, and HMAC digest. BlowFish and SHA1 are the default cipher and message digest, respectively. The EVP interface also handles PKCS#5 padding.

An important security feature in OpenVPN is the `--tls-auth` directive, which uses a pre-shared passphrase or static key to generate an HMAC key for authenticating packets in the TLS handshake sequence, enhancing protection against buffer overflows in OpenSSL's TLS implementation.

OpenVPN multiplexes the SSL/TLS session for authentication and key exchange with the encrypted tunnel data stream, ensuring SSL/TLS sees a reliable transport layer, while the IP packet forwarder sees an unreliable one. This independence between the reliability and authentication layers is crucial for efficient and secure data transmission.

The OpenVPN protocol defines a series of packet types and operations essential for establishing and maintaining a secure VPN tunnel. This includes various message types like `P_CONTROL`, `P_ACK`, and `P_DATA`, each serving specific roles in the communication process. The protocol ensures data integrity, replay protection, and seamless transition during key renegotiations, leveraging HMAC signatures and TLS sessions authenticated and initialized as described.

[Protocol details and packet formats are extensively documented in the source code, specifically in `ssl.h` within the OpenVPN repository.]