Centos7 openvpn seems not to free unused memory

[gmazzini]

An openvpn server 2.4.9 running on CentOS7 seems so eager for *free* memory as to have its openvpn-plugin-auth-pam.so plugin suddenly starting to fail authentication on any further connection (or re-connection) as soon as system *free* memory gets close to a few megabytes. Then, openvpn.log says

PLUGIN_CALL: POST /usr/lib64/openvpn/plugins/openvpn-plugin-auth-pam.so/PLUGIN_AUTH_USER_PASS_VERIFY status=1

Then only killall -HUP openvpn gets all established connections shutted down and immediately recovered, plus a fresh new bunch of "free memory". Same behaviour when openvpn-plugin-auth-pam.so is compiled from sources (in place of Debian distribution's one).
Might all that depend on some trouble in memory management?

Some more details:

• System: AWS EC2 host with Linux 3.10.0-1127.8.2.el7.x86_64, CentOS Linux release 7.8.2003 (Core).

• OpenVPN Version (from its CLI Management console):

OpenVPN 2.4.9 x86_64-redhat-linux-gnu [Fedora EPEL patched] [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [PKCS11] [MH/PKTINFO] [AEAD] built on Apr 24 2020

• Current status:

[a]. 380 clients (from openvpn-status.log, CLIENT_LIST)
[b]. running processes:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
nobody   25417  0.4  2.0 250540 77488 ?        Ss   giu18   7:04 /usr/sbin/openvpn --status /run/openvpn-status.log --status-version 2 --config openvpn.conf
root     25418  0.4  7.8 501452 299324 ?       S    giu18   5:58  \_ /usr/sbin/openvpn --status /run/openvpn-status.log --status-version 2 --config openvpn.conf

[c]. total openvpn memory (from pmap -X 25417 25418):

         Address Perm   Offset Device    Inode   Size    Rss    Pss Referenced Anonymous Swap Locked Mapping
                                               ======  =====  ===== ========== ========= ==== ======
25417:   /usr/sbin/openvpn                     250604  77556  74640      77556     73580    0      0 KB
25418:   /usr/sbin/openvpn                     753712 378532 373521     377936    372072    0      0 KB

with total [heap] ~ 389MB

[d]. free -h

              total        used        free      shared  buff/cache   available
Mem:           3,7G        606M        784M        120M        2,3G        2,7G
Swap:            0B          0B          0B

Thank you very much,
giacomo mazzini

[Gert Döring]

25418 is likely the forked-plugin-stays-root process.

My openvpn server with plugin-auth-pam does not exhibit this, and this is the first we hear about it - so it hints at a problem with one of the PAM modules loaded. Which PAM modules are you using?

I use pam_radius.so, and memory is not growing here

USER      PID   %CPU %MEM    VSZ   RSS TT  STAT STARTED          TIME COMMAND
root    60706    3.9  0.1  26832 22784  -  Ss   14Mar20    4428:17.64 /usr/local/sbin/openvpn --cd /usr/local/etc/openvpn --daemon openvpn --config server.conf
root    60707    0.0  0.0  23312 15256  -  I    14Mar20       5:48.91 /usr/local/sbin/openvpn --cd /usr/local/etc/openvpn --daemon openvpn --config server.conf

(now, this is a FreeBSD system and no Linux, but if plugin-auth-pam itself leaks memory it should be visible here too)

[Gert Döring]

Hi,

I am closing this now, as I can't do anything on a "critical" bug where the reporter isn't talking to me.

Generally speaking it seems some PAM module is leaking memory, and that is nothing OpenVPN can influence or fix. BUT: you can use the plugin-auth-pam module from git master / 2.5_beta2, which can do async/deferred authentication, which means "all PAM handling will be done in a new process for each connection".

So it does not matter how much memory the PAM stuff uses, as it's only short-lived.

(You need to add setenv deferred_auth_pam to your server config to activate the new behaviour)