Opened 3 years ago

Last modified 2 years ago

#1385 new Bug / Defect

Bridged Windows 10 Causes Sporadic Crashes

Reported by: mpfrench Owned by:
Priority: major Milestone: release 2.5.3
Component: Generic / unclassified Version: OpenVPN 2.5.0 (Community Ed)
Severity: Not set (select this one, unless your'e a OpenVPN developer) Keywords:
Cc:

Description

This bug is a continuation of bug 828 which I mistakenly thought was repaired in OpenVPN version 2.5.0. However, the frequency of the computer crashes has diminished in version 2.5.0 compared with earlier versions.

I maintain three Windows 10 servers that each run OpenVPN 2.5.0. All have exhibited this same flaw - sporadic crashes.

Each server bridges the computer Ethernet adapter to the OpenVPN TAP adapter so that remote computers can use the server's LAN for Windows networking. For the bulk of the time, the system functions as intended.

I've done extensive testing and have observed the following:
1) The crashes occur whether or not the OpenVPN services are running.

2) The crashes occur only when the server is being written to by any program, e.g. an FTPS client, while the network bridge is installed. (Each server is concurrently running the Filezilla FTPS server.)

3) The crashes never occur unless the OpenVPN TAP is bridged to the computer's Ethernet adapter.

I've captured some screen shots from one of the servers while it was being written to by a Filezilla client while the OpenVPN server was installed but deactivated, i.e., none of the OpenVPN services were running. However, the Ethernet adapter was bridged to the TAP adapter.

About 2 TB of data was being moved from the client to the server via Filezilla with the client set to automatically retry upon a failure. The screen shots were captured after the data transfer was completed.

I also perform this same data transfer without the bridge and had no crashes. So it appears that there is some unfavorable interaction between OpenVPN TAP adapter and Windows.

Attachments (13)

Server3_Critical_Events_List.jpg (207.2 KB) - added by mpfrench 3 years ago.
Server3 Critical Events List
Server3_Event_41_Kernal-Power.jpg (63.3 KB) - added by mpfrench 3 years ago.
Server3 Event_41 Kernal-Power
Server3_Minidump1.jpg (202.0 KB) - added by mpfrench 3 years ago.
Server3 Minidump 1
Server3_Minidump2.jpg (202.1 KB) - added by mpfrench 3 years ago.
Server 3 Minidump 2
Server3_Minidump3.jpg (202.4 KB) - added by mpfrench 3 years ago.
Server 3 Minidump 3
Server3_Minidump4.jpg (200.9 KB) - added by mpfrench 3 years ago.
Server 3 Minidump 4
Server3_Minidump5.jpg (202.3 KB) - added by mpfrench 3 years ago.
Server 3 Minidump 5
Server3_Realtek_Ethernet_Driver.jpg (65.7 KB) - added by mpfrench 3 years ago.
Server 3 Realtek Ethernet Driver
Server3_Network_Connections.jpg (49.4 KB) - added by mpfrench 3 years ago.
Server 3 Network Connections
Server3_Network_Bridge_Properties.jpg (64.9 KB) - added by mpfrench 3 years ago.
Server 3 Network Bridge Properties
Server3_Configure_Network_Bridge_Porperties.jpg (63.0 KB) - added by mpfrench 3 years ago.
Server 3 Configure Network Bridge Properties
minidump_20220104.jpg (215.8 KB) - added by mpfrench 2 years ago.
Minidump Analysis Screen of 4 Jan 2022
20220116_BSOD_Analysis-2_LAN_Adapters_Bridged.jpg (220.4 KB) - added by mpfrench 2 years ago.
20220116_BSOD_Analysis-2_LAN_Adapters_Bridged

Download all attachments as: .zip

Change History (22)

Changed 3 years ago by mpfrench

Server3 Critical Events List

Changed 3 years ago by mpfrench

Server3 Event_41 Kernal-Power

Changed 3 years ago by mpfrench

Attachment: Server3_Minidump1.jpg added

Server3 Minidump 1

Changed 3 years ago by mpfrench

Attachment: Server3_Minidump2.jpg added

Server 3 Minidump 2

Changed 3 years ago by mpfrench

Attachment: Server3_Minidump3.jpg added

Server 3 Minidump 3

Changed 3 years ago by mpfrench

Attachment: Server3_Minidump4.jpg added

Server 3 Minidump 4

Changed 3 years ago by mpfrench

Attachment: Server3_Minidump5.jpg added

Server 3 Minidump 5

Changed 3 years ago by mpfrench

Server 3 Realtek Ethernet Driver

Changed 3 years ago by mpfrench

Server 3 Network Connections

Changed 3 years ago by mpfrench

Server 3 Network Bridge Properties

Changed 3 years ago by mpfrench

Server 3 Configure Network Bridge Properties

comment:1 Changed 3 years ago by tct

Having the 2 TB target is sort of useful. I can retest this on Win7 hardware sometime.

@mpfrench, would you be able to test this while the bridge exists between two real ethernet cards and no TAP adapter ?

comment:2 Changed 3 years ago by Gert Döring

So this is either a bug in Win7/Win10 bridging (= would be exhibited by lan-to-lan bridging as well) or in the TAP driver. No good idea how to tackle this, though, as we're all not really skilled windows driver developers.

The general recommendation is "do not do this", though - do not use TAP mode, do not use bridging. DNS exists, AD exists, and windows networking works great over routed links - no need for bridged netbios naming requests.

comment:3 Changed 3 years ago by stipa

Would it be possible to see the stack trace? You could attach to problematic machine with windbg in advance and then when crash occurs it should break into debugger.

comment:4 Changed 3 years ago by tct

I found time to test this on Win7 OpenVPN-Server Real hardware.

Copied more than 4GB of data over a Bridged VPN with no problems at all.
I had to pull the data from a Linux server (VPN Client) to the VPN server.
I would not call this test exhaustive but it is at least something ..

Real hardware: This server is Win7 Home Premium on a laptop with 3GB RAM and a Celery chip. It is so old that it has had to have a new power supply (because the old one got ripped out too many times), the screen does not work, so uses an external monitor, and six keys are missing from the keyboard.

And it still works !

BTW: Easy-TLS a great for inlining all your X509.

Last edited 3 years ago by tct (previous) (diff)

comment:5 in reply to:  description Changed 3 years ago by tct

Replying to mpfrench:

Each server is concurrently running the Filezilla FTPS server

Free or Pro ?

comment:6 Changed 3 years ago by Gert Döring

Milestone: release 2.5release 2.5.3

Changed 2 years ago by mpfrench

Attachment: minidump_20220104.jpg added

Minidump Analysis Screen of 4 Jan 2022

comment:7 Changed 2 years ago by mpfrench

I'm am using the current version of OpenVPN - OpenVPN-2.5.5-I602-amd64.msc. The problem with Windows crashes still exists.

I did not see the last question about Filezilla until now. I was using the free version but it does not make a difference since the problem happens without using Filezilla. Just using the Windows built-in file transfer utilities will cause the crashes as well.

For instance, I have automatic nightly backups to the OpenVPN server from another computer on the LAN using the Windows command line program ROBOCOPY. The average frequency of crashes is about every 4 days. This backup procedure does not use OpenVPN. It is a straight Windows-to-Windows file transfer.

I've uploaded the last minidump analysis screen, minidump_20220104.jpg.

comment:8 Changed 2 years ago by mpfrench

In an effort to determine whether or not Windows was the source of the crashes when using a bridged connection, I installed a second LAN adapter. It turned out that both the original LAN adapter and the recent addition were made by Realtek and used the same driver software - PCIe GbE Family Controller. I used the latest Realtek driver for the testing. I then bridged both LAN adapters. The OpenVPN adapters were not part of the bridge.

I then forced the transfer of about 250 GB of data to the computer running the bridge. During the second run of this test, the computer running the bridge crashed. See the minidump analysis labeled 20220116_BSOD_Analysis-2_LAN_Adapters_Bridged.jpg.

This analysis shows the culprit to be bridge.sys, a Windows system component, with the exact same fault "SYSTEM_THREAD_EXCEPTION_NOT_HANDLED" as I observed while previously running with an OpenVPN TAP adapter bridged to a LAN adapter.

It appears as though the crashes that I'm observing while running a bridge are due strictly to Windows itself. Does anyone have another conclusion?

Also, if it really is due to Windows, how do we report this to M$?

Changed 2 years ago by mpfrench

20220116_BSOD_Analysis-2_LAN_Adapters_Bridged

comment:9 Changed 2 years ago by stipa

I think the best way for you to proceed would be indeed to contact Microsoft. According to this https://community.osr.com/discussion/comment/303646/#Comment_303646 they will bill 500$ in advance, and if the problem appears to be bug in Windows, they will refund the money.

https://support.microsoft.com/contactus
https://support.microsoft.com/en-us/topic/microsoft-professional-support-pay-per-incident-faq-575821bc-17bb-7484-4935-334c5437639f

Note: See TracTickets for help on using tickets.