packet loss during duo push request

[gustavosv]

First of all, I'm newbie here and I didn't can found solution to my problem, searching before writing this :)
My server runs fine and my clients can connect without problems.
Openvpn is installed on a server (OPNSENSE) and I'm doing Authentication certificate + ldap (AD user) + MFA duo in a centOS7 through the plugin duo.

Well, when some users are connected to VPN and other agent wants connect to, the VPN tunnels of the all others clients (previously connected and their tunnels stablished) freezes. I can see how ping to server throught VPN lose atleast 3-5 packets. It's weird.

During the analysis I realized that the problem occurs when openvpn sends the push request(duo-phone). As long as the client does not accept the request(duo), everyone in the tunnel loses a packet.

we can also say problem may be related to a plugin call auth-user-pass-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify

Attach server conf:

cat /var/etc/openvpn/server1.conf
dev ovpns1
verb 5
dev-type tun
dev-node /dev/tun1
writepid /var/run/openvpn_server1.pid
script-security 3
daemon
ping-timer-rem
persist-key
proto tcp-server
cipher AES-128-CBC
auth SHA256
up /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkup
down /usr/local/etc/inc/plugins.inc.d/openvpn/ovpn-linkdown
local xxx.xxx.xxx.xxx
client-disconnect "/usr/local/etc/inc/plugins.inc.d/openvpn/attributes.sh server1"
tls-server
server 192.168.xxx.0 255.255.255.0
client-config-dir /var/etc/openvpn-csc/1
auth-user-pass-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify  user 'LDAP_VPN' 'true' 'server1'" via-env
tls-verify "/usr/local/etc/inc/plugins.inc.d/openvpn/ovpn_auth_verify tls 'internal-ca' 1"
lport 1194
management /var/etc/openvpn/server1.sock unix
push "route 10.xxx.0.0 255.xxx.0.0"
push "route 172.xxx.0.0 255.xxx.0.0"
push "dhcp-option DNS 10.xxx.96.100"
push "dhcp-option WINS 10.xxx.96.101"
ca /var/etc/openvpn/server1.ca
cert /var/etc/openvpn/server1.cert
key /var/etc/openvpn/server1.key
dh /usr/local/etc/dh-parameters.2048.sample
tls-auth /var/etc/openvpn/server1.tls-auth 0
reneg-sec 0
reneg-bytes 0
tcp-queue-limit 512

Attach cient conf:

dev tun
persist-key
proto tcp-client
cipher AES-128-CBC
auth SHA256
client
resolv-retry infinite
reneg-sec 0
remote xxxxxxx.com.br 1194 tcp
lport 0
verify-x509-name "C=BR, ST=SP, L=SP, O=TI, emailAddress=test@test.com, CN=internal-ca" subject
remote-cert-tls server
auth-user-pass
pkcs12 XXXXXXXXXXXXXXXX.p12
tls-auth XXXXXXXXXXXXXXXXXXXX.key 1
verb 5

Duo auth config

[root@ldapproxy01 ~]# cat /opt/duoauthproxy/conf/authproxy.cfg
; Complete documentation about the Duo Auth Proxy can be found here:
; https://duo.com/docs/authproxy_reference

; MAIN: Include this section to specify global configuration options.
; Reference: https://duo.com/docs/authproxy_reference#main-section
[main]
debug=true

; CLIENTS: Include one or more of the following configuration sections.
; To configure more than one client configuration of the same type, append a
; number to the section name (e.g. [ad_client2])

[ad_client]
host=10.XXX.96.101
host_2=10.XXX.96.100
service_account_username=XXXXXXXXXXXX
service_account_password=XXXXXXXXXXXXXXX
search_dn=DC=TI,DC=intranet

; SERVERS: Include one or more of the following configuration sections.
; To configure more than one server configuration of the same type, append a
; number to the section name (e.g. radius_server_auto1, radius_server_auto2)

[ldap_server_auto1]
client=ad_client
ikey=XXXXXXXXXXXXXXXXXXXX
skey=XXXXXXXXXXXXXXXXX
api_host=XXXXXXXXXXXXXXXXX
failmode=secure
exempt_primary_bind=false
exempt_ou_1=OU=Users_Application,DC=TI,DC=intranet
[root@ldapproxy01 ~]#

ovpn_auth_verify

#!/bin/sh

if [ "$1" = "tls" ]; then
        /usr/local/bin/php /usr/local/etc/inc/plugins.inc.d/openvpn/tls-verify.php -d "$2" "$3"
else
        # Single quoting $password breaks getting the value from the variable.
        # XXX I really don't like going through openssl for this...
        password=$(echo -n "${password}" | /usr/local/bin/openssl enc -base64 | sed -e 's/=/%3D/g')
        username=$(echo -n "${username}" | /usr/local/bin/openssl enc -base64 | sed -e 's/=/%3D/g')

        /usr/local/bin/php /usr/local/etc/inc/plugins.inc.d/openvpn/auth-user.php "$username" "$password" "$common_name" "$3" "$2" "$4"
fi

exit $?

Client Log:
in this line the request of the duo is sent to mobile, if the user does not accept the request on the cell phone, everyone in the tunnel loses package.
Wed Mar 17 16:16:51 2021 TCP_CLIENT link remote: [AF_INET]XXX.XXX.XXX.XXX:1194

Wed Mar 17 16:16:48 2021 OpenVPN 2.4.9 x86_64-w64-mingw32 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] built on Apr 16 2020
Wed Mar 17 16:16:48 2021 Windows version 6.2 (Windows 8 or greater) 64bit
Wed Mar 17 16:16:48 2021 library versions: OpenSSL 1.1.1f  31 Mar 2020, LZO 2.10
Wed Mar 17 16:16:50 2021 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Wed Mar 17 16:16:50 2021 TCP/UDP: Preserving recently used remote address: [AF_INET]XXX.XXX.XXX.XXX.145:1194
Wed Mar 17 16:16:50 2021 Attempting to establish TCP connection with [AF_INET]XXX.XXX.XXX.XXX:1194 [nonblock]
Wed Mar 17 16:16:51 2021 TCP connection established with [AF_INET]XXX.XXX.XXX.XXX:1194
Wed Mar 17 16:16:51 2021 TCP_CLIENT link local (bound): [AF_INET][undef]:0
Wed Mar 17 16:16:51 2021 TCP_CLIENT link remote: [AF_INET]XXX.XXX.XXX.XXX:1194
Wed Mar 17 16:17:05 2021 [internal-ca] Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XXX:1194
Wed Mar 17 16:17:06 2021 open_tun
Wed Mar 17 16:17:06 2021 TAP-WIN32 device [Conexão Local] opened: \\.\Global\{XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX}.tap
Wed Mar 17 16:17:06 2021 Notified TAP-Windows driver to set a DHCP IP/netmask of XXX.XXX.XXX.XXX/XXX.XXX.XXX.XXX on interface {xxxxxxxxxxxxxx} [DHCP-serv: XXX.XXX.XXX.XXX, lease-time: 31536000]
Wed Mar 17 16:17:06 2021 Successful ARP Flush on interface [14] {xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx}
Wed Mar 17 16:17:11 2021 Initialization Sequence Completed

Server Log:
in this line the request of the duo is sent to mobile, if the user does not accept the request on the cell phone, everyone in the tunnel loses package.
Wed Mar 17 16:16:41 2021 TCP_CLIENT link remote: [AF_INET]XXX.XXX.XXX.XXX:1194
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_GUI_VER=OpenVPN_GUI_11

Mar 17 16:16:44 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:59960 SIGUSR1[soft,connection-reset] received, client-instance restarting
Mar 17 16:16:44 OPNsense openvpn[9299]: TCP/UDP: Closing socket
Mar 17 16:16:50 OPNsense openvpn[9299]: MULTI: multi_create_instance called
Mar 17 16:16:50 OPNsense openvpn[9299]: Re-using SSL/TLS context
Mar 17 16:16:50 OPNsense openvpn[9299]: Control Channel MTU parms [ L:1623 D:1170 EF:80 EB:0 ET:0 EL:3 ]
Mar 17 16:16:50 OPNsense openvpn[9299]: Data Channel MTU parms [ L:1623 D:1450 EF:123 EB:406 ET:0 EL:3 ]
Mar 17 16:16:50 OPNsense openvpn[9299]: Local Options String (VER=V4): 'V4,dev-type tun,link-mtu 1571,tun-mtu 1500,proto TCPv4_SERVER,keydir 0,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-server'
Mar 17 16:16:50 OPNsense openvpn[9299]: Expected Remote Options String (VER=V4): 'V4,dev-type tun,link-mtu 1571,tun-mtu 1500,proto TCPv4_CLIENT,keydir 1,cipher AES-128-CBC,auth SHA256,keysize 128,tls-auth,key-method 2,tls-client'
Mar 17 16:16:50 OPNsense openvpn[9299]: TCP connection established with [AF_INET]XXX.XXX.XXX.XXX:53182
Mar 17 16:16:50 OPNsense openvpn[9299]: TCPv4_SERVER link local: (not bound)
Mar 17 16:16:50 OPNsense openvpn[9299]: TCPv4_SERVER link remote: [AF_INET]XXX.XXX.XXX.XXX:53182
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 TLS: Initial packet from [AF_INET]XXX.XXX.XXX.XXX:53182, sid=0a394370 37da2150
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 VERIFY SCRIPT OK: depth=0, C=BR, ST=SP, L=SP, O=TI, emailAddress=XXXXXXXXXXXX, CN=USERTESTE
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 VERIFY OK: depth=0, C=BR, ST=SP, L=SP, O=TI, emailAddress=XXXXXXXXXXXXXXX, CN=USERTESTE
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_VER=2.4.9
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_PLAT=win
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_PROTO=2
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_NCP=2
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_LZ4=1
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_LZ4v2=1
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_LZO=1
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_COMP_STUB=1
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_COMP_STUBv2=1
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_TCPNL=1
Mar 17 16:16:51 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 peer info: IV_GUI_VER=OpenVPN_GUI_11
Mar 17 16:17:05 OPNsense openvpn[23509]: user 'USERTESTE' authenticated using 'LDAP - VPN'
Mar 17 16:17:05 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 TLS: Username/Password authentication succeeded for username 'USERTESTE'
Mar 17 16:17:05 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 Control Channel: TLSv1.3, cipher TLSv1.3 TLS_AES_256_GCM_SHA384, 2048 bit RSA
Mar 17 16:17:05 OPNsense openvpn[9299]: XXX.XXX.XXX.XXX:53182 [USERTESTE] Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XXX:53182
Mar 17 16:17:05 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 MULTI_sva: pool returned IPv4=XXX.XXX.XXX.XXX, IPv6=(Not enabled)
Mar 17 16:17:05 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 MULTI: Learn: XXX.XXX.XXX.XXX-> USERTESTE/XXX.XXX.XXX.XXX:53182
Mar 17 16:17:05 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 MULTI: primary virtual IP for USERTESTE/XXX.XXX.XXX.XXX:53182: XXX.XXX.XXX.XXX
Mar 17 16:17:06 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 PUSH: Received control message: 'PUSH_REQUEST'
Mar 17 16:17:06 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 SENT CONTROL [USERTESTE]: 'PUSH_REPLY,route 10.XXX.0.0 255.0.0.0,push-continuation 2' (status=1)
Mar 17 16:17:06 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 SENT CONTROL [USERTESTE]: 'PUSH_REPLY,route 15.XXX.0.0 255.128.0.0,dhcp-option DOMAIN XXXXXXXXXXXXX,dhcp-option DNS XXX.XXX.XXX.XXX,dhcp-option DNS XXX.XXX.XXX.XXX,dhcp-option WINS XXX.XXX.XXX.XXX,dhcp-option WINS 10.116.96.101,route XXX.XXX.XXX.XXX,topology net30,ifconfig XXX.XXX.XXX.XXX XXX.XXX.XXX.XXX,peer-id 0,cipher AES-256-GCM,push-continuation 1' (status=1)
Mar 17 16:17:06 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 Data Channel: using negotiated cipher 'AES-256-GCM'
Mar 17 16:17:06 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 Data Channel MTU parms [ L:1551 D:1450 EF:51 EB:406 ET:0 EL:3 ]
Mar 17 16:17:06 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Mar 17 16:17:06 OPNsense openvpn[9299]: USERTESTE/XXX.XXX.XXX.XXX:53182 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key

[tct]

This is probably all the same problem:
​https://forums.openvpn.net/viewtopic.php?f=4&t=31988#p98520

[Gert Döring]

Yeah, the forum thread has most of the answers already.

That said, there are a few different possible approaches

• convert the script to a plugin, and have that plugin run asynchronously (deferred)

• use Mozilla's helper plugin which can do this for you :-) see here: ​https://github.com/mozilla-it/openvpn_defer_auth (for OpenVPN it is a "plugin", but all it does is "run a script, and report back the script status to OpenVPN").

• wait for 2.6, which will be able to run scripts asynchronously as well - but that's more for completeness. With 2.4 or 2.5, go for the existing and well-tested deferred plugin API.

I will be closing this as "wontfix", as we can't do anything on the OpenVPN side about it (there is no plan to go for multithreaded behaviour in the foreseeable timeframe, even though lots of us have toyed with the idea). It is documented behaviour, workarounds exist. And yes, it would be nicer otherwise, but this is what it is.