Possible concurrency issue with auth-pam plugin blocking multiple users

[jnchi]

Reproduction Steps

On the OpenVPN server, install and configure pam_aad following the instructions, then start the server and tail the logs,

tail -f /var/log/openvpn/*.log /var/log/auth.log

Setup at least two OpenVPN clients, then attempt to connect the first client.

After attempting to authenticate the first client via the device code, attempt to connect the second client.

Note: The second client never receives an email address.

See also:

• ​https://github.com/CyberNinjas/pam_aad/issues/39
• ​https://community.openvpn.net/openvpn/ticket/201

[jnchi]

Archive of configuration files used for testing

[Selva Nair]

The pam plugin included in the OpenVPN sources (auth_pam) does work with multiple clients. So I would think the problem may be in pam_aad -- just guessing, I haven't looked at the sources of that plugin.

If Azure AD supports LDAP you can test using auth_pam in OpenVPN plus pam_ldap included in PAM -- if that works you know the issue is with pam_aad. Such a setup does work with on-premise AD.

Does pam_aad do deferred auth? If it does, comparison with auth_pam is not a good one as the latter does not exercise that code path and there could be a problem there.

[jnchi]

Thanks for the response. As one of the authors of pam_aad, I am also auditing the code to ensure there are no bugs on this end, but the only issues I've encountered are when we are trying to authenticate with OpenVPN. I have no issues with OpenSSH or any other services.

To my knowledge Azure AD does not support LDAP; in fact, the only similarities it shares with active directory is the name itself.

What is meant by deferred auth? Basically, pam_aad requests a JWT token using the devicecode oauth2 flow, and then makes a request against the Microsoft Graph API to check user and group privileges.

[Selva Nair]

The logs you posted show four plugin calls which all return, so calling out to the plugin multiple times does work. Two calls are with username "eh1" which fail and two with "lrimage" which succeed. Reusing the same name (lrimage) replaces the old connection with a new one which may not be what you want.

Also the plugin calls appear to take 30 to 40seconds to complete which maybe leading to some timing
issues as the client seems to be sending repeated PUSH_REQUESTs even after the server has sent PUSH_REPLY. As all calls come from same IP, and usernames are reused, hard to say whether they are all same client or understand what the real issue is. Post matching server and client logs at verb=4 with two independent clients connecting in parallel or one after the other -- we may find something in there.

As for deferred auth: looking at the configs you posted, I see pam_aad.so is a PAM module, not an openvpn plugin, and you are interfacing with OpenVPN using the stock auth-pam plugin. The latter does not support deferred auth, so that comment is irrelevant.

That said, I think you should use a plugin that does deferred auth: as OpenVPN is single threaded authentication scripts and plugins would block and traffic to all clients stall until the auth-verify call returns. Deferred auth avoids this by promptly returning to OpenVPN with OPENVPN_PLUGIN_FUNC_DEFERRED and later writing auth status to a control file. Particularly important when the authentication process involves slow network calls.

See include/openvpn-plugin.h and sample/sample-plugins/defer/simple.c in the source repo. This may be beyond the scope of pam_aad, however.

[jnchi]

I setup OpenVPN on a virtual machine, and then I setup an OpenVPN client in a docker container. I then spun up a second client via Docker, and attempted to authenticate with a different username; hence the eh1, and lramage usernames from a single IP.

I'll work on getting the requested logs posted.

That is correct. Sorry for the miscommunication.

Is it possible to modify the default auth-pam plugin for deferred authentication? That would be most ideal.

[jnchi]

I believe I found a security vulnerability in the deferred auth sample plugin. I tried sending an encrypted email to security@openvpn.net, but apparently the gpg key provided expired last year.

[Gert Döring]

The deferred auth sample plugin is... very poor code, and somewhere on my list of things to clean up.

In the meantime I have added deferred authentication support to our plugin auth-pam plugin. This is not part of the 2.4 distribution, but is in git master or release/2.5, and will ship with 2.5.0 release. The calling convention is the same, so the 2.5 plugin can be used with 2.4.x openvpn servers just fine.

To enable deferred auth, when you have a recent-enough plugin, you need to add a

setenv deferred_auth_pam 1

to your openvpn config (otherwise the plugin will keep using the old synchronous code path).

[Gert Döring]

Since we never heard back, and I think I have provided a good solution, I'll proceed to close this ticket now.