Bridged Windows 10 Causes Sporadic Crashes

[mpfrench]

This bug is a continuation of bug 828 which I mistakenly thought was repaired in OpenVPN version 2.5.0. However, the frequency of the computer crashes has diminished in version 2.5.0 compared with earlier versions.

I maintain three Windows 10 servers that each run OpenVPN 2.5.0. All have exhibited this same flaw - sporadic crashes.

Each server bridges the computer Ethernet adapter to the OpenVPN TAP adapter so that remote computers can use the server's LAN for Windows networking. For the bulk of the time, the system functions as intended.

I've done extensive testing and have observed the following:
1) The crashes occur whether or not the OpenVPN services are running.

2) The crashes occur only when the server is being written to by any program, e.g. an FTPS client, while the network bridge is installed. (Each server is concurrently running the Filezilla FTPS server.)

3) The crashes never occur unless the OpenVPN TAP is bridged to the computer's Ethernet adapter.

I've captured some screen shots from one of the servers while it was being written to by a Filezilla client while the OpenVPN server was installed but deactivated, i.e., none of the OpenVPN services were running. However, the Ethernet adapter was bridged to the TAP adapter.

About 2 TB of data was being moved from the client to the server via Filezilla with the client set to automatically retry upon a failure. The screen shots were captured after the data transfer was completed.

I also perform this same data transfer without the bridge and had no crashes. So it appears that there is some unfavorable interaction between OpenVPN TAP adapter and Windows.

[mpfrench]

Server3 Critical Events List

[mpfrench]

Server3 Event_41 Kernal-Power

[mpfrench]

Server3 Minidump 1

[mpfrench]

Server 3 Minidump 2

[mpfrench]

Server 3 Minidump 3

[mpfrench]

Server 3 Minidump 4

[mpfrench]

Server 3 Minidump 5

[mpfrench]

Server 3 Realtek Ethernet Driver

[mpfrench]

Server 3 Network Connections

[mpfrench]

Server 3 Network Bridge Properties

[mpfrench]

Server 3 Configure Network Bridge Properties

[tct]

Having the 2 TB target is sort of useful. I can retest this on Win7 hardware sometime.

@mpfrench, would you be able to test this while the bridge exists between two real ethernet cards and no TAP adapter ?

[Gert Döring]

So this is either a bug in Win7/Win10 bridging (= would be exhibited by lan-to-lan bridging as well) or in the TAP driver. No good idea how to tackle this, though, as we're all not really skilled windows driver developers.

The general recommendation is "do not do this", though - do not use TAP mode, do not use bridging. DNS exists, AD exists, and windows networking works great over routed links - no need for bridged netbios naming requests.

[stipa]

Would it be possible to see the stack trace? You could attach to problematic machine with windbg in advance and then when crash occurs it should break into debugger.

[tct]

I found time to test this on Win7 OpenVPN-Server Real hardware.

Copied more than 4GB of data over a Bridged VPN with no problems at all.
I had to pull the data from a Linux server (VPN Client) to the VPN server.
I would not call this test exhaustive but it is at least something ..

Real hardware: This server is Win7 Home Premium on a laptop with 3GB RAM and a Celery chip. It is so old that it has had to have a new power supply (because the old one got ripped out too many times), the screen does not work, so uses an external monitor, and six keys are missing from the keyboard.

And it still works !

BTW: ​Easy-TLS a great for inlining all your X509.

[tct]

Replying to mpfrench:

Each server is concurrently running the Filezilla FTPS server

Free or Pro ?

[mpfrench]

Minidump Analysis Screen of 4 Jan 2022

[mpfrench]

I'm am using the current version of OpenVPN - OpenVPN-2.5.5-I602-amd64.msc. The problem with Windows crashes still exists.

I did not see the last question about Filezilla until now. I was using the free version but it does not make a difference since the problem happens without using Filezilla. Just using the Windows built-in file transfer utilities will cause the crashes as well.

For instance, I have automatic nightly backups to the OpenVPN server from another computer on the LAN using the Windows command line program ROBOCOPY. The average frequency of crashes is about every 4 days. This backup procedure does not use OpenVPN. It is a straight Windows-to-Windows file transfer.

I've uploaded the last minidump analysis screen, minidump_20220104.jpg.

[mpfrench]

In an effort to determine whether or not Windows was the source of the crashes when using a bridged connection, I installed a second LAN adapter. It turned out that both the original LAN adapter and the recent addition were made by Realtek and used the same driver software - PCIe GbE Family Controller. I used the latest Realtek driver for the testing. I then bridged both LAN adapters. The OpenVPN adapters were not part of the bridge.

I then forced the transfer of about 250 GB of data to the computer running the bridge. During the second run of this test, the computer running the bridge crashed. See the minidump analysis labeled 20220116_BSOD_Analysis-2_LAN_Adapters_Bridged.jpg.

This analysis shows the culprit to be bridge.sys, a Windows system component, with the exact same fault "SYSTEM_THREAD_EXCEPTION_NOT_HANDLED" as I observed while previously running with an OpenVPN TAP adapter bridged to a LAN adapter.

It appears as though the crashes that I'm observing while running a bridge are due strictly to Windows itself. Does anyone have another conclusion?

Also, if it really is due to Windows, how do we report this to M$?

[mpfrench]

20220116_BSOD_Analysis-2_LAN_Adapters_Bridged

[stipa]

I think the best way for you to proceed would be indeed to contact Microsoft. According to this ​https://community.osr.com/discussion/comment/303646/#Comment_303646 they will bill 500$ in advance, and if the problem appears to be bug in Windows, they will refund the money.

​https://support.microsoft.com/contactus
​https://support.microsoft.com/en-us/topic/microsoft-professional-support-pay-per-incident-faq-575821bc-17bb-7484-4935-334c5437639f