non-interruptible loop in windows dns resolution failure

[Gert Döring]

• win7 VM with no IPv6 (no interface has v6, so win7 disables v6 altogether)
• git master 6417a6f8a0
• connecting with "--proto udp6"
• run from console window

-> result is a "DNS resolution fails, retrying" (because windows will refuse to even lookup a v6 record if "there is no v6 in the system!") endless loop, neither ctrl-c nor f1...f4 work.

Didn't try from GUI, was too annoyed at my VM setup... but should be reproduceable easy enough.

The endless loop happens on Windows as well, but ctrl-c works - here's a linux log:

Mon Dec 14 19:42:29 2015 RESOLVE: Cannot resolve host address: v4only.greenie.net:1194 (Name or service not known)
Mon Dec 14 19:42:29 2015 Could not determine IPv4/IPv6 protocol
Mon Dec 14 19:42:29 2015 SIGUSR1[soft,init_instance] received, process restarting
Mon Dec 14 19:42:29 2015 Restart pause, 5 second(s)
Mon Dec 14 19:42:34 2015 Control Channel Authentication: tls-auth using INLINE static key file
Mon Dec 14 19:42:34 2015 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Mon Dec 14 19:42:34 2015 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Mon Dec 14 19:42:34 2015 RESOLVE: Cannot resolve host address: v4only.greenie.net:1194 (Name or service not known)
Mon Dec 14 19:42:34 2015 RESOLVE: Cannot resolve host address: v4only.greenie.net:1194 (Name or service not known)
Mon Dec 14 19:42:34 2015 Could not determine IPv4/IPv6 protocol
Mon Dec 14 19:42:34 2015 SIGUSR1[soft,init_instance] received, process restarting
Mon Dec 14 19:42:34 2015 Restart pause, 5 second(s)
^CMon Dec 14 19:42:35 2015 SIGINT[hard,init_instance] received, process exiting

[Gert Döring]

thanks :)

[Selva Nair]

Replying to cron2:

• win7 VM with no IPv6 (no interface has v6, so win7 disables v6 altogether)
• git master 6417a6f8a0
• connecting with "--proto udp6"
• run from console window

-> result is a "DNS resolution fails, retrying" (because windows will refuse to even lookup a v6 record if "there is no v6 in the system!") endless loop, neither ctrl-c nor f1...f4 work.

Didn't try from GUI, was too annoyed at my VM setup... but should be reproduceable easy enough.

The endless loop happens on Windows as well, but ctrl-c works

A quick question: is this with Xen, KVM, something else? I've seen ctrl-c sometimes ignored on windows 10 even on physical hardware, though a few tries always work.

Please try ctrl-break as well. Although it was added along with ctrl-c, the former is delivered by windows as a signal, the latter as a key-press like f1..f4, while running from console.

[Selva Nair]

Failed DNS resolution can loop can become non-interruptible even on Linux: as a test use an unsused ip number as the nameserver in resolv.conf and start a connection. Nearly impossible to break out of it by SIGINT or SIGTERM.

There are a number of places in socket.c where sig_info->signal_received is assigned to SIGUSR1, overwriting previous value which could be SIGTERM or SIGINT (e.g., line socket.c:1919 which appears to be the culprit in this case). Note that sig_info here is a pointer to signinfo_static and its members are volatile. They can change when signals interrupt.

Interestingly its the hard SIGTERM/SIGINT that is easily lost in this case -- a SIGTERM simulated through the management doesn't get noticed until the loop restarts and goes back to init.c. So it survives this blatant over-write of signal_received in socket.c

[Gert Döring]

see also #311 which is the same issue but got forgotten in the meantime *sigh*

[Gert Döring]

This needs to be re-tested and then either be closed or fixed.

[Gert Döring]

I think I've bumped into this with 2.6_beta2 - in certain conditions, OpenVPN just ignores incoming SIGTERM (seems to be "when in SIGUSR1 restart wait").

[Selva Nair]

We need a revamp of how signals are implemented: use posix signal (sigaction) so that signals can be blocked and a priority order can be enforced etc.

I worked on this years back but dropped the ball. If there is enough interest in pursuing this approach I can resurrect it:

See: ​https://github.com/OpenVPN/openvpn/commit/7e5d775227e6d304ce24d7505da9332f405ee4f3

Or here is a summary from 2018 (some things may be outdated)
`
Fix signal handling issues

Currently signal received is directly modified in many places in the code,
leading to loss of signals, low priority signals overwriting higher priority
ones etc.

• Set all signals using functions like register_signal

• Add a function register_signal_si to help setting of signals when only the pointer to the signal_info struct is available.

• Allow only a higher or equal priority signal to overwirte an already registered but yet to be processed signal. The signals in increasing order of priority are SIGUSR2, SIGUSR1, SIGHUP, SIGTERM, SIGINT.

• Use posix signals (sigaction) to properly block signals while in the handler etc.

• Collect windows signals even when management is not available. Currently Windows signals cannot interrupt openvpn_sleep unless the management interface is in use: the latter forces management_event_loop_n_seconds() in place of sleep().

`

[Selva Nair]

I guess the patch linked above would be too much, too late for 2.6? What about a band-aid fix that adds signal priority but does not extensively re-write sig.c?

[Gert Döring]

Discussion on the signal issues was summarized in

​https://github.com/OpenVPN/openvpn/issues/205

and fixed in a number of patches from Selva Nair that went into 2.6.0 - basically doing what was suggested above. The "POSIX sigaction" is still open for review and more time for cross-platform testing.

I am not sure if we want to backport the signal handling changes to release/2.5 ("it qualifies as bugfix"), because it's quite a large changeset to be able to cleanly address this particular effect (pending signals are overwritten by soft-signals at getaddrinfo() erorrs). Given the lack of sustained yelling, this seems to be a more infrequent annoyance mostly - and since *I* opened this particular ticket, I consider it solved sufficiently for my needs, I never run old versions of OpenVPN on Windows.

Thanks, Selva :-)