openvpn client gives up instead of retrying when proxy server is slow

[justusranvier]

I have an instance of OpenVPN connecting to a server via a Tor (SOCKS) proxy. Due to the nature of Tor, sometimes a connection will drop and openvpn will attempt to reconnect, and sometimes the reconnection attempts will timeout. When this happens, the process will exit and require a manual restart.

The client should retry the operation instead of exiting. As it is now, maintaining a persistent openvpn connection over Tor requires an external script to continually poll the client process and manually restart it if it's not running.

Sep 06 06:28:32 [openvpn] Socket Buffers: R=[87380->131072] S=[16384->131072]
Sep 06 06:28:32 [openvpn] Attempting to establish TCP connection with [AF_INET]<scrubbed>:9050 [nonblock]
Sep 06 06:28:33 [openvpn] TCP connection established with [AF_INET]<scrubbed>:9050
Sep 06 06:28:38 [openvpn] recv_socks_reply: TCP port read timeout expired: Operation now in progress (errno=115)
Sep 06 06:28:38 [openvpn] SIGTERM[soft,init_instance] received, process exiting

[grarpamp]

OpenVPN:

openvpn [2.3_git:master/30e358e5de352c8d+] [opts] --socks-proxy 127.0.0.1 9050
...
Attempting to establish TCP connection with [AF_INET]127.0.0.1:9050 [nonblock]
TCP connection established with [AF_INET]127.0.0.1:9050
recv_socks_reply: TCP port read timeout expired: Operation now in progress (errno=36)
TCP/UDP: Closing socket
SIGTERM[soft,init_instance] received, process exiting

FreeBSD:

36 EINPROGRESS Operation now in progress. An operation that takes a long
time to complete (such as a connect(2)) was attempted on a non-
blocking object (see fcntl(2)).

​http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=657964

Some more searching will show this problem dates back to at least 2009.
I've not seen any fix/patch posted anywhere yet.
Searches show the number of users wanting to use this configuration
and having problems with it are increasing.

OpenVPN should be aware that SOCKS5 proxies can be very slow and
adjust it's timings accordingly/configurably when using a proxy.

[Gert Döring]

We did some serious work with the connect/retry logic in master in the past few weeks, but I don't think this will change things for you - in the logs we can see that the connection was established successfully, but then the socks server itself was (too) slow in responding. So the *connection* retry logic is already finished.

Right now all I can do is acknowledge that this might pose a problem, but since we're (as always) fairly short on developer resources, we can't promise when we'll find time to dig into this and come up with a fix.

If someone else can track this down and can post a patch, I'm happy to review and merge...

[anon_tor_user]

Patch to allow configuration of the socks proxy timeout

[anon_tor_user]

After using the patch above to change the socks proxy timeout to be 60 seconds the TCP timeout issue mentioned in the original report was resolved. I'd like to commend the OpenVPN maintainers for keeping the code so well organized. This was my first time looking at the code, and I was able to patch it up in a matter of hours and everything went exactly as expected.

Full disclosure: I was not able to get the following configuration working:
Me -> Socks proxy (Tor) -> VPN Server -> Internet

My firewall DROPs all inbound and outbound packets for the physical interface except DNS requests and Tor traffic. All traffic is allowed in/out of the loopback interface. I don't fully understand how the routing works, so I don't know if I will have to open up tun0 or not. I tried allowing all traffic on tun0, however that did not seem to resolve the problem.

[badon]

I've written up an increasingly sophisticated script to handle this issue. To be reliable, sometimes Tor needs to be restarted. Also, if there is a temporary network outage (perhaps caused by maintenance or a power outage), I want OpenVPN and Tor to continue attempting to reconnect endlessly. I'm also considering adding features to reboot network hardware too. I'm considering putting my software on Github, and maybe porting it to other languages and platforms (it's currently for Windows).

I've put a lot more effort into this than I originally planned. Would it be straightforward to add the feature to OpenVPN so it can endlessly reconnect, in addition to the ability to adjust the SOCKS proxy timeout? If that feature is going to be implemented, then maybe I should focus my efforts elsewhere.

I recommend upgrading the priority of this ticket to "major", because the ability to continuously attempt to reconnect in networking software is both important and very common (maybe universal). It's absence in OpenVPN under some circumstances is thus conspicuous, ought to be fixed eventually. Note that this ticket is already 12 months old at the time I'm writing this.

[Gert Döring]

As I mentioned, we're unfortunately way too short on developer resources - so we fix those bits that we personally experience (and/or add code that we personally need), and the rest is "as time permits". I don't think one of us is using Tor right now, or extensively using any sort of proxy, so this is not personally affecting us...

If the OpenVPN bits can be fixed by a quick patch, we're happy to review and merge, but actually digging into this (especially since the git master code is likely quite different from 2.3 here, so master might or might not have the same issues, and in any case, the fix will look different) will not happen "any time soon now". Sorry.

[amg1127]

I am currently unable to increase proxy timeout if OpenVPN version 2.3.10 connects through a SOCKS proxy. The option --socks-proxy-timeout is not recognized and I can't use --http-proxy-timeout with --socks-proxy:

Options error: Unrecognized option or missing parameter(s) in [CMD-LINE]:1: socks-proxy-timeout (2.3.10)
Use --help for more information.

Options error: --http-proxy not specified but other http proxy options present
Use --help for more information.

The timeout seems to be hardcoded here: ​https://github.com/OpenVPN/openvpn/blob/master/src/openvpn/socks.c#L323

I wish the patch above could be reviewed and merged in OpenVPN code.

[plaisthos]

There is already a better patch to the fix this problem on the mailing list (​https://sourceforge.net/p/openvpn/mailman/message/34542459/). Unfortenately it is still pending review and has not been commited yet.

[David Sommerseth]

Please re-test this against the latest git master development version. This have a commit which is related to connection timeouts, and may very well help resolve this issue. Currently the changes seems to be too massive to be considered for a 2.3 release. But the git master is the coming 2.4 release.

commit f2134b7bea37df15756c599b94f16d4bffafbbd6
Author: Arne Schwabe <arne@rfc2549.org>
Date:   Sat Jun 11 16:43:15 2016 +0200

    Remove http-proxy-timeout, socks timeout and set default of server-poll-timeout to 120s
    
    With this change all timeouts before the first packet from the OpenVPN
    server are unified into the server-poll-timeout option.
    
    The default of 120s has been chosen to be a safe value is larger as it is
    larger the sums of the old small timeouts.
    
    V3: fix some whitespace/typos problems
    Acked-by: Gert Doering <gert@greenie.muc.de>
    Message-Id: <1465656195-12722-1-git-send-email-arne@rfc2549.org>
    URL: http://article.gmane.org/gmane.network.openvpn.devel/11899
    
    Signed-off-by: Gert Doering <gert@greenie.muc.de>

If no response is received before 2.4 reaches the alpha release stage, this issue will be closed as fixed.

[David Sommerseth]

Considering this issue fixed.

[Gert Döring]

In a few more words - these two patches basically solve this in git master / 2.4

commit 2011b8324feca30df753a4a0a116d37c04742520
Author: Arne Schwabe <arne@…>
Date: Fri Jun 24 14:27:10 2016 +0200

Remove http-proxy-retry and socks-proxy-retry.

commit f2134b7bea37df15756c599b94f16d4bffafbbd6
Author: Arne Schwabe <arne@…>
Date: Sat Jun 11 16:43:15 2016 +0200

Remove http-proxy-timeout, socks timeout and set default of server-poll-timeout to 120s

With this change all timeouts before the first packet from the OpenVPN
server are unified into the server-poll-timeout option.

[rugen]

Openvpn2(v2.4.6) of MacPorts? generates an error with socks-proxy 127.0.0.1 9050 and
socks-proxy-retry.

WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Initialization Sequence Completed
[*.opengw.net] Inactivity timeout (--ping-restart), restarting
SIGUSR1[soft,ping-restart] received, process restarting
Restart pause, 5 second(s)
WARNING: No server certificate verification method has been enabled.  See http://openvpn.net/howto.html#mitm for more info.
TCP/UDP: Preserving recently used remote address: [AF_INET]127.0.0.1:9050
Socket Buffers: R=[131072->131072] S=[131072->131072]
Attempting to establish TCP connection with [AF_INET]127.0.0.1:9050 [nonblock]
TCP connection established with [AF_INET]127.0.0.1:9050
recv_socks_reply: TCP port read timeout expired: Operation timed out (errno=60)
SIGUSR1[soft,init_instance] received, process restarting
Restart pause, 5 second(s)

[Gert Döring]

please do not just post to unrelated tickets that have been long closed - if you have a proxy problem, it's very unlikely related to this one.

Please open a new ticket, and do not (never!) remove the time stamps from the log file - especially when talking about timeouts and "too slow", if you remove the timestamps, nobody can tell what happened. Also, include your config file and more detailed descriptions about your environment, like "what proxy are you using" - this could well be a user error (telling OpenVPN it's a SOCKS proxy while it's a HTTP proxy, etc.)

[ValdikSS]

This issue is fixed neither in 2.5.9 nor in 2.6.0 or git master.

Here's a link to hard-coded 5 second timeout.

​https://github.com/OpenVPN/openvpn/blob/b999466418dddb89721be5455f03bacd9b221b1d/src/openvpn/socks.c#L194

[Antonio Quartulli]

Replying to ValdikSS:

This issue is fixed neither in 2.5.9 nor in 2.6.0 or git master.

Here's a link to hard-coded 5 second timeout.

​https://github.com/OpenVPN/openvpn/blob/b999466418dddb89721be5455f03bacd9b221b1d/src/openvpn/socks.c#L194

This comment is unrelated to this ticket.
This comment refers to OpenVPN having a 5 seconds timeout, which is not always long enough and cause the connection to restart while the proxy is still doing its thing.

Please open a new ticket for this specific issue.