Different Windows versions have different kernel-mode signing options:

* Windows 7/8/8.1/Server 2012r2

* Cross-signing

* WHQL-certified (HCR)

* Windows 10 desktop

* Attestation signing

* WHQL-certified (HLK)

* Windows Server 2016/2019

* Attestation signing

* WHQL-certified (HLK)

In this article we focus on HLK testing.

**NOTE:** it **is not required** to pass the HLK tests just to get a driver that loads on Windows Server 2016/2019. An attestation-signed driver is good enough. This claim contradicts the "official" Microsoft documentation but trust me, it is true. Our installers have attestation-signed drivers and no Windows Server 2016/2019 users have complained. Apparently that particular piece of MS documentation was written at a time when MS was _planning_ to require WHQL-certified drivers for Windows Server 2016+, then backpedaled and forgot to update the documentation.

Sgstair patched tap-windows6 to pass the HLK tests. [His work](https://github.com/sgstair/tap-windows6/tree/hlkwork) has now been merged to the upstream tap-windows6 project

According to practical testing done by [wintun](https://www.wintun.net/) developers it is possible to get a code signature that is valid for all Windows 10 platforms using the following HLK clients:

* Windows Server 2019 Desktop (64-bit)

* Currently the [Static Tools Logo Test](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/6ab6df93-423c-4af6-ad48-8ea1049155ae) has to run on "Desktop" Server variant

* Windows Server 2019 Core (64-bit)

* The [Operate in Server Core test](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/ac3eb111-539a-4b7b-93f2-d542bd8a2135) needs to run on "Core" server variant

* Windows 10 desktop (32-bit)

Wintun was able to pass HLK testing without any *physical* HLK clients. But because Wintun advertised itself as a virtual device it had a narrower scope and had to pass fewer HLK tests (~50 in total) than tap-windows6 (68 tests with filters applied).

For the HLK controller you can use a virtualized (Virtualbox, VMware) Windows Server 2016 or 2012r2 instance.

For tap-windows6 testing a support machine is also needed. To be on the safe side use the same OS version and build as the HLK client.

There are some additional requirements for tap-windows6 that stem from generic [LAN testing prerequisites](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/lan-testing-prerequisites):

* HLK client needs at least 4 virtual processor cores (unverified) for Windows Server certification

* HLK clients need to be physical computers, not virtualized (unverified)

Also try to get a quality fast network switch. Slow switches can cause issues on some of the tests. However, a cheap 10€ switch may end up working just fine.

For HLK software installation please refer to the official MS documentation, check out [puppet-hlk](https://github.com/Puppet-Finland/puppet-hlk/) and [puppet-hlk_tap6_openvpn](https://github.com/Puppet-Finland/puppet-hlk_tap6_openvpn) or try out the [Windows Virtual Hardware Lab Kit](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/getstarted/getstarted-vhlk).

The version of HLK you need to install depends on the version of Windows you're attempting certify as described in [Microsoft documentation](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/). To check Windows version from Powershell do:

For multimachine tests you need to name some of the devices on the HLK test client (that actually runs the tests) and support machine (second machine that is connected “back to back” through the VPN). The non-VPN interface through which the nodes communicate with the HLK controller should be named "MessageDevice" and the tap-windows6 adapter on the support machine should be named "SupportDevice0". Details in [LAN testing prerequisites](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/lan-testing-prerequisites).

You also have to enable test signing so that you can load unsigned ("to be tested") drivers on both HLK client and support machine:

Then you need to install the HLK client software from the HLKInstall SMB share on the controller.

You also need the automatically generated test certificate ("WDKTest*") from the tap-windows6 build machine to the Windows certificate store on the HLK clients. After all this you can install the test-signed driver without signature errors.

The "Run tests" in HLK fail consistently unless the tap-windows6 adapter has an IPv6 gateway address. This can be resolved by a simple bridged peer to peer OpenVPN setup, where interface settings are configured statically outside of OpenVPN.

The first steps are:

* Install the latest OpenVPN 2.x on the HLK client and support machine

* Install the same, test-signed (to-be-HLK-tested) tap-windows6 driver on the HLK clients

* Configure static IP, netmask, gateway, etc. for the TAP interface

* Disable Windows Firewall for the TAP adapter / Private networks. This is not strictly necessary, but saves time.

Then generate a shared secret with "openvpn --genkey" so that you can use it in the OpenVPN config. The OpenVPN configuration file for HLK client and support machines can be identical except for the "remote" settings:

The above setup is symmetric in the sense that neither node is a client or a server, and either one can initiate the connection. This is similar to what is described in the [static key mini-howto](https://openvpn.net/community-resources/static-key-mini-howto/).

To verify OpenVPN connectivity launch it from an administrator Powershell session first on both machines:

If you see that connection was established try to ping the VPN IPs (IPv4 and IPv6) of the other party from both ends. If that succeeds, you can stop OpenVPN and let OpenVPNService manage it from there on:

Some have had more luck with the legacy service (OpenVPNServiceLegacy).

When you have created a Project the next step is to select what to test. Go to "Selection" tab and select "TAP-Windows Adapter V9" of the HLK client (not support) machine.

In the HLK Studio's Project tab you will see a category called "Product Types" at the right, like in [here](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/hlk-studio). When you create a new project the "Product Types" section will be empty, and you need to make it says LAN, or you end up doing all the testing work and only get certified as an "Other Driver".

HLK product types are listed in the [product type matrix](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/hlk-product-type-matrix). The way to add the "LAN" product type is to go to the "Selection" Tab, select "Device Manager" on the left, then right click on the TAP adapter and add the feature for Device.Network.LAN.PM (power management). HLK then notices that all features for the "LAN" product type are met and it adds the "LAN" product type to the project.

Make sure to get the HLK Hardware Compatibility Playlists (on the main HLK download page), and apply the one for the correct context, e.g. HLK Version 1809 CompatPlaylist x64 Server.xml. The playlist narrows down the list of tests to the set required to get an HLK certification, removes some extra stress/failure verification type tests designed to help find driver crashes. There’s a “Load Playlist” option in the tests panel in the HLK studio app that you can use.

It seems required/useful to add HLK clients to the pool first, then the support machine. Otherwise when you select the driver in "Device manager" tab HLK will assume that the HLK support machine has the driver under test. This *may* create problems down the line.

Then when you go to schedule the multimachine test, you can change the “role” dropdown to the support machine, and select the support machine (should be there if you have two ready machines in the pool).

Some of the non-NDIS tests that require “a working network connection” can just be given a valid IPV6 address on the VPN network and they will be happy with that. Whenever you see the “WDTFREMOTESYSTEM” parameter when scheduling a test, set it to an IPv6 address the system can ping over the VPN link. This could be the ipv6 address of the “support” system, or it could be some other arbitrary system on the VPN. This might not be necessary if the device under test has a working IPv6 gateway address that points to, say, support machine's tap adapter interface. But wearing belt and suspenders is not a bad idea when running HLK tests.

This test tests plugging and unplugging the (virtual) ethernet cable.

For this test you have two options:

* Start and stop the service when running this to make the link start and stop

* Note that you can't do this from Powershell prompt of the DTMLLUAdminUser which you may end up being in. You can still use the graphical services management, or possibly use an elevated Powershell prompt.

* Use Tapdiag to change link state on the test system

In either case follow the interactive messagebox prompts. You may need to detach and attach the "cable" twice.

Use Tapdiag to enable 802.1Q on both machines before running the NDIS QoS test (2c_Priority).

The process for using tapdiag is:

1. tapdiag /enable # Sets a registry key that enables the .tapdiag endpoint in the driver

1. Restart the TAP driver, reboot or disable/enable in device manager. May need to stop the openvpn service to avoid a reboot – Reconnect VPN

1. tapdiag /link:[on|off] # Use to unplug/plug ethernet cable

1. tapdiag /setq:[on|off|always] # to set the 802.1Q handling. Driver now disables 802.1Q by default.

Note that the tapdiag configuration is runtime only – if you reboot the test machine, you will lose the 802.1Q state.

This will definitely fail on slower machines. It worked fine on 1 year old i7 systems.

The *E2EPerf* test seems to fail if the test adapter and the support adapter have different link speeds. This is visible from the logs of "Run NDISTest Client (no verifier)":

This is a problem if the support machine has an old tap-windows6 driver version that claims to be 100Mbps whereas the device under test ("DUT") is advertising itself as 1Gbps device. Resolve by installing the correct (1Gbps) tap-windows6 driver version to the support machine as well.

This test can if you happen to have even one Network device that does not have a functional driver. The failure will then happen at the "Run NDISTest Client" phase when it tries to stop the (tap-windows6) driver:

More details are available in the log file *ndistest.htm*:

The only hint here that tap-windows6 itself might not be to blame is the phrase "all network devices". In my case on Windows Server 2019 Datacenter Evaluation the problem was a Wireless LAN adapter whose driver was not functioning properly. The fix was to disable Wireless LAN altogether from BIOS. Installing a working driver would probably have solved the problem as well.

Once you get past these failure, at the end of the test !OpenvpnService might not re-establish a link. WHQL seems to get the OpenVPN service into a bad state, just restart the OpenVPN service and the test will pass. The test waits a few minutes for the link to come back up.

This test may fail at "Run NDISTest Client" -> "Restore protocol bindings" phase:

While the error message is not the same as with 1c_FaultHandling and with 1c_Registry, the cause seems to be the same, as well as the remedy: ensure that you don't have any disfunctional network device (drivers).

This test has the same issues as 1c_FaultHandling, above. The same remedies apply.

Sometimes this test will hit a breakpoint in the NDIS test code. The breakpoint seems harmless and complained about some packets not getting confirmed. If you don't connect a kernel debugger this will cause a !BugCheck and the test will fail. If this happens connect a kernel debugger and rerun the test. When the breakpoint is hit press (l) Always Ignore and the test will pass.

You may also get away with just rerunning the test until it passes.

This test may fail due to broken network drive mappings. The hints are available in "Infrastructure -> Execution Logs -> WttEa.log:

If this happens you should see a network share that is unavailable:

To resolve, unmount the network drive:

The [Static Tools Logo Test](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/6ab6df93-423c-4af6-ad48-8ea1049155ae) checks for the presence of a Driver Verification Log (DVL) log file as described in the [official MS documentation](https://docs.microsoft.com/en-us/windows-hardware/drivers/develop/creating-a-driver-verification-log). The verification log does not have to come from the same *build* as the one you install to the HLK clients, but it almost certainly has to be based on the exact same *source code*.

Tap-windows6 buildsystem runs code analysis automatically for x64 Release builds, so build that variant as the first step. Test analysis files are under directory *tap-windows6\src\x64\Release* named as *<sourcefile>.nativecodeanalysis.xml*. Results of the code analysis are not enough, as you also need to [Creating a log file for Static Driver Verifier](https://docs.microsoft.com/en-us/windows-hardware/drivers/develop/creating-a-log-file-for-static-driver-verifier).

Creating the Static Driver Verifier logs is a simple process once you know it. Launch a *x64 Native Tools command prompt* that comes with Visual Studio 2019. Then

Note that you need to "cd" to the "src" directory. You also need to provide the full path to the vcxproj file. The SDV tests will take quite a bit of time so don't wait holding your breath.

Once they pass you need to merge the results of SDV and codeanalysis. You can do this from Visual Studio 2019 from the "Extensions" -> "Driver" menu, or using the command-line:

The end result is a file called *tap0901.DVL.XML*, which you should copy to *C:\DVL* on one of the HLK clients, after which the Static Tools Logo Test should pass.

An issue that occurs in many tests and which *looks* serious, but is actually not that, is the error at "Copy downlevel NDISTest binaries", where it starts to look for non-existing files from the Controller SMB share:

If you mount the share manually on the HLK client you'll notice that either the entire directories or individual .sys files are missing indeed:

While this step fails, it seems that it *can* fail yet the test as a whole can pass. The failure reason icon (see [here](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/troubleshooting-windows-hlk-test-failures)) in this case should be "Canceled", i.e. "A user canceled the test, or a task has been canceled because the preceding task failed."

If you upgrade HLK software on both controller and client/support you may end up with a situation where you can't put any of the clients into "Ready" state. You may be able to resolve this by removing the machines from the pool, and deleting them from the default pool, and then restarting HLKSvc service on the clients.

This can result in test errors like "Expected minimum of 237 packets but we received 200 packets". The test allows one second after all the sends have been completed for all the packets to be received.

Disabling verbose log printing in the server makes it more reliable.

Packets can (rarely) be reordered in flight, which causes an assertion in the test driver. Hints are errors such as

* "1 total breakpoints were hit in the protocol driver while this test was executing"

* "Out of order indication"

* "Dropped indications"

HLK doesn't complain if some packets are lost, instead these errors are raised when out of order packets are received.

The OpenVPN architecture has some inherent race conditions that can cause reordering of packets. This has happened 2 or 3 times over dozens of runs, and does cause a test failure, but a rerun should pass.

You may encounter failures in the "Address Change" test. It’s a combination of a few factors:

* OpenVPN client sets the network link status around a second before the server actually starts forwarding any packets to the client. This is quite possibly an order of operations bug in openvpn.

* That would be well and good, but the address change test has extremely aggressive timings

* It starts sending packets almost immediately once the link status comes up, and stops listening for packets less than a second after that.

This failure seems to be related to newer OpenVPN versions.

Logs for reference:

SIGNTASK : SignTool error : No certificates were found that met all the given criteria.

PS> Remove-Item Cert:\CurrentUser\My\91047502F73D5410C106E05CEC8A990219810FBE

PS> Get-Childitem Cert:\Currentuser\My\F1E725722C5BB56757B6D261D958425869213089|Export-Certificate -Filepath C:\users\samuli\opt\wdktest-samuli.cer

Get-Childitem .\wdktest-samuli.cer| Import-Certificate -Certstorelocation cert:\LocalMachine\TrustedPublisher

```

The puppet-hlk_tap6_openvpn module handles the import part automatically with Powershell DSC.

HLK controller logs problems mostly to event logs, so when tests are failing in an unusual fashion you can possibly find out why in the event viewer. When you have issue putting an HLK client into "Ready" state, for example, have a look at the event log.

The HLK tests fortunately provide tons of logging visible for HLK Studio. Check sections "Error", "Task log", "Infrastructure" in the context menu for the test in question to see what you can find.

It is possible to get HLK into strange states. For example:

* HLK clients / support machines do not get into "Ready" state

* Launching two-machine tests fail with duplicate database key errors

Some things you can try to get past these errors:

* Restart the "hlksvc" Windows service

* Reboot HLK nodes (controller, clients, support machine)

* Reinstall HLK client software and reboot HLK client

* Create a new HLK project

* Create a new pool and move HLK client machines there

* Delete HLK clients from the pool

The are known to work sometimes, but so far I have not been able to establish any logic here.

Do not change the default browser on the HLK controller, or you may be unable to view the HLK test logs directly from the HLK controller. This is known to happen if the default browser is set to Firefox, but other non-Internet Explorer browsers may be affected.

Changing the TAP interface type to be a virtual adapter in the INF file does not seem to work. It seems to mess up the NDIS tests which looked for a device that advertised as physical to assign a SupportDevice. Maybe this is something we can eventually work with Microsoft to fix.

Reboots seems to happen randomly during test setup. This can be a nuisance if you are monitoring the services window or the network connections

window. Helpful to make shortcuts to these so they can easily be opened.

Installing HLK software automatically opens ports in the Windows firewall for HLK traffic. In case HLK controller and HLK clients are not in the same switch some firewall (e.g. EC2 security group rules) might block HLK traffic. Here is a reference for the ports which need to be open for HLK tests to work:

* OpenVPN peer (udp/1194) <-> OpenVPN peer (udp/1194)

* HLK clients -> HLK controller tcp/1771 (HLK Server Receiver Port)

* HLK clients -> HLK controller tcp/1782 (HLKSvc Receiver Port)

* HLK clients -> HLK controller tcp/445 (HLKInstall Samba share)

* HLK controller -> HLK clients tcp/1771 (HLK Server Receiver Port)

Outbound traffic is assumed to be unrestricted. If not, adjust egress rules accordingly. Also note that IPv6 traffic needs to flow properly in the OpenVPN virtual network as HLK tests require IPv6.

* [Three-Plus Years Later… Driver Signing Still Baffles](https://www.osr.com/blog/2019/06/03/three-plus-years-later-driver-signing-still-baffles)

* [Device.Network tests](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/device-network-tests) (reference)

* [Windows Hardware Certification blog](https://techcommunity.microsoft.com/t5/Windows-Hardware-Certification/bg-p/WindowsHardwareCertification)

* [Troubleshooting Windows HLK](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/troubleshooting-windows-hlk)

* [tapdiag](https://github.com/sgstair/tapdiag): a tool that is used to manipulate tap-windows6 at runtime for some HLK tests

* [puppet-hlk](https://github.com/Puppet-Finland/puppet-hlk/): Puppet module for setting up HLK controllers and HLK clients

* [puppet-hlk_tap6_openvpn](https://github.com/Puppet-Finland/puppet-hlk_tap6_openvpn): Puppet module for setting up OpenVPN and network interface configurations for HLK testing

* [Windows Virtual Hardware Lab Kit](https://docs.microsoft.com/en-us/windows-hardware/test/hlk/getstarted/getstarted-vhlk)

* [http://timr.probo.com/wd3/071503/NDISTest.htm Testing Network Drivers with the NDIS Test Tool]