iOS: NIP crash on EVENT WAIT after sleep on wifi

[hunterx1]

I have so far had 2 occurrences of a case where the OpenVPN connect client would disconnect with the last line in the log being EVENT WAIT. No vpn status indicator is displayed. When opening OpenVPN, the connection slider is in the disconnected state, and the status is disconnected. So far the issue seems random. Listed as major due to disconnect without warning.

2018-02-02 21:20:58 EVENT: CONNECTED 123456@???.com:1198 (xxx.xxx.xxx.xx) via /UDPv4 on NetworkExtensionTUN/10.23.10.10/ gw=/
2018-02-02 21:29:11 OS Event: SLEEP
2018-02-02 21:29:11 EVENT: PAUSE
2018-02-02 21:36:52 OS Event: WAKEUP
2018-02-02 21:36:55 RESUME TEST: Internet:ReachableViaWiFi/-R t------
2018-02-02 21:36:55 STANDARD RESUME
2018-02-02 21:36:55 EVENT: RESUME
2018-02-02 21:36:55 EVENT: RECONNECTING
2018-02-02 21:36:55 Contacting [xxx.xxx.xxx.xx]:1198/UDP via UDP
2018-02-02 21:36:55 EVENT: WAIT

[hunterx1]

Found in iPhone analytics:
Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000
VM Region Info: 0 is not in any region. Bytes before following region: 4295983104

REGION TYPE START - END [ VSIZE] PRT/MAX SHRMOD REGION DETAIL
UNUSED SPACE AT START

--->

TEXT 00000001000f8000-00000001000fc000 [ 16K] r-x/r-x SM=COW ...NIP.appex/NIP

Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [0]
Triggered by Thread: 0

[Antonio Quartulli]

this clearly looks like a crash. Thanks a lot for the extra log! It seems a NULL pointer dereference, but it's not easy to understand where this is happening. Should you find a way to reliably reproduce it, please let us know!

[hunterx1]

unfortunately that would be incredibly hard because of how random it is. The only reliable bit of info I can offer is that it fails the exact moment the phone is woken up from sleep. This has now occurred 3 times. There doesn't appear to be a pattern for the crash frequency. crash 1 20180131 16:06:20.78, crash 2 20180202 21:36:56.19, crash 3 20180203 07:36:26.44. All times in GMT -8. All crashes listed occurred while on wifi. All 3 have the same crash report in iPhone analytics with bug type 109, and the same exact crash info as above with the 0 address.

iPhone 6s 128 running iOS 11.2.5.

OpenVPN Connect settings:
seamless tunnel ON
Connect Via "Any Network"
Reconnect on wakeup ON
Protocol "Adaptive"
Compression "Full"
Connection timeout "None"
Network state detection "Active"
IPV6 "Disabled"
Force AES-CBC ciphersuites OFF
Minimum TLS version "TLS 1.2"
Allow MD5 algorithm in TLS OFF
Google DNS fallback OFF
Layer 2 reachability ON
SSL log level "0"
no proxy config

ovpn file info (default file from provider with keys omitted):
client
dev tun
remote address.com
1198 udp
remote address.com
502 tcp
resolv-retry-infinite
nobind
persist-key
persist-tun
setenv CLIENT_CERT 0
tls-client
comp-lzo
verb 1
reneg-sec 0

EDIT: added bit of info, I see other crash reports starting with OpenVPN Connect 1.2.5 related to invalid memory addresses, but before 1.2.7, the address was not null. After 1.2.7, all the addresses in the crash reports are null.

[nullbandit]

Adding my logs to the case. @ 18:06 NIP Crashed in thread 0 with seg 11.

{"app_name":"NIP","timestamp":"2018-02-04 18:06:55.74 -0800","app_version":"1.2.7","slice_uuid":"603536be-bad5-37b5-8815-a0a7f5873f5e","adam_id":590379981,"build_version":"4","bundleID":"net.openvpn.connect.app.NIP","share_with_app_devs":false,"is_first_party":false,"bug_type":"109","os_version":"iPhone OS 11.2.5 (15D60)","incident_id":"E014E55E-D070-4294-8DC4-D7E00C49BB4A","name":"NIP"}
Incident Identifier: E014E55E-D070-4294-8DC4-D7E00C49BB4A
CrashReporter Key:   2db4c13e9f8f48aaf2b0e698d96226aab01c23a5
Hardware Model:      iPhone9,1
Process:             NIP [637]
Path:                /private/var/containers/Bundle/Application/FDDE6E7D-3CFB-4B1F-AFA5-56E9265672E8/OpenVPN.app/PlugIns/NIP.appex/NIP
Identifier:          net.openvpn.connect.app.NIP
Version:             4 (1.2.7)
Code Type:           ARM-64 (Native)
Role:                Unspecified
Parent Process:      launchd [1]
Coalition:           net.openvpn.connect.app.NIP [531]


Date/Time:           2018-02-04 18:06:54.8772 -0800
Launch Time:         2018-02-03 10:19:04.5201 -0800
OS Version:          iPhone OS 11.2.5 (15D60)
Baseband Version:    3.42.00
Report Version:      104

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000
VM Region Info: 0 is not in any region.  Bytes before following region: 4298375168
      REGION TYPE                      START - END             [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                 0000000100340000-0000000100344000 [   16K] r-x/r-x SM=COW  ...NIP.appex/NIP

Termination Signal: Segmentation fault: 11
Termination Reason: Namespace SIGNAL, Code 0xb
Terminating Process: exc handler [0]
Triggered by Thread:  0

Filtered syslog:
None found

Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   NIP                           	0x00000001003d3af0 0x100340000 + 604912
1   NIP                           	0x00000001003ecc70 0x100340000 + 707696
2   NIP                           	0x00000001003ecc70 0x100340000 + 707696
3   NIP                           	0x00000001003ea548 0x100340000 + 697672
4   libdispatch.dylib             	0x000000018585ea14 0x18585d000 + 6676
5   libdispatch.dylib             	0x000000018586cdcc 0x18585d000 + 64972
6   libdispatch.dylib             	0x000000018585ea14 0x18585d000 + 6676
7   libdispatch.dylib             	0x000000018586b698 0x18585d000 + 59032
8   CoreFoundation                	0x0000000185e87344 0x185d9c000 + 963396
9   CoreFoundation                	0x0000000185e84f20 0x185d9c000 + 954144
10  CoreFoundation                	0x0000000185da4c58 0x185d9c000 + 35928
11  Foundation                    	0x00000001867d9594 0x1867cd000 + 50580
12  Foundation                    	0x000000018682b56c 0x1867cd000 + 386412
13  libxpc.dylib                  	0x0000000185b50e10 0x185b3d000 + 81424
14  libxpc.dylib                  	0x0000000185b52efc 0x185b3d000 + 89852
15  Foundation                    	0x0000000186a05bb4 0x1867cd000 + 2329524
16  PlugInKit                     	0x000000018b3c5d48 0x18b3b3000 + 77128
17  PlugInKit                     	0x000000018b3c590c 0x18b3b3000 + 76044
18  PlugInKit                     	0x000000018b3c5d6c 0x18b3b3000 + 77164
19  Foundation                    	0x00000001869e6cf0 0x1867cd000 + 2202864
20  libdyld.dylib                 	0x00000001858c456c 0x1858c3000 + 5484

Down below the threat 0 stack on NIP clearly shows the CSPR is null

Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x0000000000000000   x1: 0x0000000129dd8ce0   x2: 0x00000000ffffffff   x3: 0x0000000000000001
    x4: 0x0000000000001903   x5: 0x00000000ffffffff   x6: 0x0000000000000000   x7: 0x0000000000000004
    x8: 0x0000000000000000   x9: 0x000000016fc5e0a0  x10: 0x00000001003d3ae8  x11: 0x0000000000000000
   x12: 0x0000000000000001  x13: 0x0000009000000000  x14: 0x139d7700139d7700  x15: 0x0000000000000000
   x16: 0x00000000000001de  x17: 0x00000000ffffffff  x18: 0xfffffff01f0dd260  x19: 0x000000016fc5e0a8
   x20: 0x00000001b724dc60  x21: 0x00000001b7249140  x22: 0x0000000000000000  x23: 0x0000000129e2dbd0
   x24: 0x00000001b724dc60  x25: 0x0000000000000003  x26: 0x0000000129e2dbd0  x27: 0x0000000000000000
   x28: 0x0000000002ffffff   fp: 0x000000016fabe9f0   lr: 0x00000001003ecc70
    sp: 0x000000016fabe9f0   pc: 0x00000001003d3af0 cpsr: 0x00000000

@ 19:30 OpenVPN app wokeup to crash because SPRINGBOARD terminated the app and crashed in thread 0.

{"app_name":"OpenVPN","timestamp":"2018-02-04 19:30:28.00 -0800","app_version":"1.2.7","slice_uuid":"b3131dff-8559-38a8-8107-eb9e125e3845","adam_id":590379981,"build_version":"4","bundleID":"net.openvpn.connect.app","share_with_app_devs":false,"is_first_party":false,"bug_type":"109","os_version":"iPhone OS 11.2.5 (15D60)","incident_id":"54AA4793-E312-4543-B036-7302CC0A1D9E","name":"OpenVPN"}
Incident Identifier: 54AA4793-E312-4543-B036-7302CC0A1D9E
CrashReporter Key:   2db4c13e9f8f48aaf2b0e698d96226aab01c23a5
Hardware Model:      iPhone9,1
Process:             OpenVPN [212]
Path:                /private/var/containers/Bundle/Application/FDDE6E7D-3CFB-4B1F-AFA5-56E9265672E8/OpenVPN.app/OpenVPN
Identifier:          net.openvpn.connect.app
Version:             4 (1.2.7)
Code Type:           ARM-64 (Native)
Role:                Foreground
Parent Process:      launchd [1]
Coalition:           net.openvpn.connect.app [352]


Date/Time:           2018-02-04 19:30:27.7841 -0800
Launch Time:         2018-01-31 14:49:59.5807 -0800
OS Version:          iPhone OS 11.2.5 (15D60)
Baseband Version:    3.42.00
Report Version:      104

Exception Type:  EXC_CRASH (SIGKILL)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note:  EXC_CORPSE_NOTIFY
Termination Reason: Namespace SPRINGBOARD, Code 0xdeadfa11
Termination Description: SPRINGBOARD, power down shortcut |  | ProcessVisibility: Background | ProcessState: Suspended
Triggered by Thread:  0

Filtered syslog:
None found

Thread 0 name:  Dispatch queue: com.apple.main-thread
Thread 0 Crashed:
0   libsystem_kernel.dylib        	0x00000001859d3568 0x1859d2000 + 5480
1   libsystem_kernel.dylib        	0x00000001859d33e0 0x1859d2000 + 5088
2   CoreFoundation                	0x0000000185e87108 0x185d9c000 + 962824
3   CoreFoundation                	0x0000000185e84cd4 0x185d9c000 + 953556
4   CoreFoundation                	0x0000000185da4c58 0x185d9c000 + 35928
5   GraphicsServices              	0x0000000187c50f84 0x187c46000 + 44932
6   UIKit                         	0x000000018f4fd5c4 0x18f48a000 + 472516
7   OpenVPN                       	0x000000010246fb10 0x102468000 + 31504
8   libdyld.dylib                 	0x00000001858c456c 0x1858c3000 + 5484

[nullbandit]

To add to previous log,

OpenVPN Connect App crashes on devices running tcp bound tunnel not on "udp" bound tunnels.

But NIP crashes at same offset 0x604912 with "CSPR" pointer being NULL irrespective of TCP/UDP tunnel.

In all previous revisions starting from v1.2.5 when NIP crashed CSPR pointer was intact and showed address 0x80000000 and it crashed in various places in the code.

My best guess, there is possible bug in fast changing/with poor reachability to network (/System/Library/Frameworks?/CFNetwork.framework) in the past many have seen this behavior in CFNetwork framework if that helps.

I would suggest, If others are testing please test with "wifi assist" turned off.

[hunterx1]

Replying to nullbandit:

To add to previous log,

OpenVPN Connect App crashes on devices running tcp bound tunnel not on "udp" bound tunnels.

But NIP crashes at same offset 0x604912 with "CSPR" pointer being NULL irrespective of TCP/UDP tunnel.

In all previous revisions starting from v1.2.5 when NIP crashed CSPR pointer was intact and showed address 0x80000000 and it crashed in various places in the code.

My best guess, there is possible bug in fast changing/with poor reachability to network (/System/Library/Frameworks?/CFNetwork.framework) in the past many have seen this behavior in CFNetwork framework if that helps.

I would suggest, If others are testing please test with "wifi assist" turned off.

Interesting. I do use Wifi Assist, but my WIFI signal is really strong, and only the LTE is low signal. My phone gets -64dbm in the place it failed last. In my case, it failed connecting on UDP as well (Adaptive setting). I do not see any of those "OpenVPN" entries, just the NIP ones. I will try turning OFF wifi assist to see if that helps.

EDIT:
while on the topic of low signal, i was at a location yesterday with no wifi, and barely any cell coverage at all. the strongest network I could connect to was 3G, and it was barely enough to get notifications and texts though, yet the VPN worked. I may have gotten lucky, but the phone switched to wifi and connected without issues when I returned home...

[nullbandit]

Replying to hunterx1:

while on the topic of low signal, i was at a location yesterday with no wifi, and barely any cell coverage at all. the strongest network I could connect to was 3G, and it was barely enough to get notifications and texts though, yet the VPN worked. I may have gotten lucky, but the phone switched to wifi and connected without issues when I returned home...

Do you have VoLTE enabled ( Cellular -> Enable LTE Voice & Data ) and Data Roaming on ?

This setting "might" contribute to that issue as well (not sure yet). Baseband behavior on LTE during roaming and VoLTE is kind of bumpy ATM. Some issues are attributed by MVNO providers during the switching, band hopping and throttling, that could interfere as well. I am still testing with options, but seems like there is a bug in NIP somewhere where buffers are crashing, devs might be able to provide insight.

[hunterx1]

yes, I have LTE Voice & Data enabled, but roaming is disabled. I already disabled WiFi? Assist.

[nullbandit]

Another factor that "might" contribute Is "Background App Refresh" . I keep it toggled to off to stop traffic leaking when VPN accidentally breaks or the provider drops my connection. Its hard to predict network behavior, When its Background App refresh is toggled to off and the interface band switching occurs (LTE -> 3G, 3G -> LTE, LTE->wifi, WiFi? -> LTE) there could be packet buffering issues but at least from privacy perspective its safer imo.

[hunterx1]

Replying to nullbandit:

Another factor that "might" contribute Is "Background App Refresh" . I keep it toggled to off to stop traffic leaking when VPN accidentally breaks or the provider drops my connection. Its hard to predict network behavior, When its Background App refresh is toggled to off and the interface band switching occurs (LTE -> 3G, 3G -> LTE, LTE->wifi, WiFi? -> LTE) there could be packet buffering issues but at least from privacy perspective its safer imo.

I always have Background App Refresh disabled, for various reasons.

EDIT: I do not see OpenVPN in the list of apps that use the Background App Refresh feature. I enabled it temporarily to see if the list would refresh, and it did not.

[hunterx1]

Wifi Assist was OFF, and I just had a disconnect. Connection tried to reconnect as TCP that time.

EDIT: I have re-enabled Wifi assist, and have enabled Background App Refresh (with all apps turned off manually). OpenVPN does not appear in the list (YET?).

[Antonio Quartulli]

Replying to hunterx1:

EDIT: added bit of info, I see other crash reports starting with OpenVPN Connect 1.2.5 related to invalid memory addresses, but before 1.2.7, the address was not null. After 1.2.7, all the addresses in the crash reports are null.

Yes, let's not mix crashes from apps before v1.2.7. The latter comes with a broad internal restructure. The 0x0 address you see is a distinct issue, not to be confused with previous crashes.

Thanks all for the information provided. I am relaying all these details internally.
They should help to better understand the issue.

About "Background App Refresh": I think you'll never see OpenVPN there because the way it works is different from other background apps. OpenVPN has a separate process that handles the VPN connection (separated from the GUI) and it works in a sort of privileged manner. It's the way Apple has designed its Network Extension API.

But NIP crashes at same offset 0x604912

This might be an important detail.

Thanks again, I'll get back when I'll have an update.

[Antonio Quartulli]

A change that attempts to fix this issue will be shipped with the next 1.2.8(1) beta

[Antonio Quartulli]

1.2.8 has been approved and is being released to the AppStore?? as we speak. I am finally closing this ticket. Please feel free to re-open it if the issue is not yet solved.