Changes between Version 49 and Version 50 of HLKTesting


Ignore:
Timestamp:
06/20/19 12:26:08 (3 years ago)
Author:
Samuli Seppänen
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • HLKTesting

    v49 v50  
    2929According to practical testing done by [https://www.wintun.net/ wintun] developers it is possible to get a code signature that is valid for all Windows 10 platforms using the following HLK clients:
    3030
    31 * HLK controller: Windows Server 2016
    32 * HLK clients
    33  * Windows Server 2019 (64-bit)
    34  * Windows Server 2019 core (64-bit)
    35  * Windows 10 desktop (32-bit)
     31* Windows Server 2019 (64-bit)
     32* Windows Server 2019 core (64-bit)
     33* Windows 10 desktop (32-bit)
    3634
    3735Wintun was able to pass HLK testing without any ''physical'' HLK clients. But due to wintun's narrower scope it had to pass much fewer HLK tests (~50 in total) than tap-windows6.
    3836
    39 For tap-windows6 testing a couple of extra nodes are needed:
    40 
    41 * OpenVPN server (for "Run tests" in HLK, see below)
    42 * Support machine: required by some of the HLK tests
    43 
    44 = Lan testing prerequisites =
     37Both Windows Server 2016 and 2012r2 should work fine as the HLK controller. The controller can be virtualized (e.g. Virtualbox or VMWare).
     38
     39For tap-windows6 testing a support machine is also needed. To be on the safe side use the same OS version and build as the HLK client.
    4540
    4641There are some additional requirements for tap-windows6 that stem from generic [https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/lan-testing-prerequisites LAN testing prerequisites]:
     
    5146Also remember to rename the network devices ("MessageDevice", "SupportDevice0") as described [https://docs.microsoft.com/en-us/windows-hardware/test/hlk/testref/lan-testing-prerequisites LAN testing prerequisites].
    5247
     48Also try to get a quality fast network switch. Slow switches can cause issues on some of the tests. However, a cheap 10€ switch may end up working just fine.
     49
    5350= Installing HLK software =
    5451
     
    6562Installation of HLK client software automatically enables test signing mode in Windows. Tap-windows6 build system supports test-signing the driver automatically. You need to put the automatically generated test certificate to the Windows certificate store on the HLK clients. After that you can install the test-signed driver without signature errors.
    6663
    67 = Preparing HLK studio/controller =
    68 
    69 == Set product type ==
    70 
    71 In the HLK Studio's Projet tab you will see a category called "Product Types" at the right, like in [https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/hlk-studio here]. When you create a new project the "Product Types" section will be empty, and you need to make it says LAN, or you end up doing all the testing work and only get certified as an "Other Driver".
    72 
    73 HLK product types are listed in the [https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/hlk-product-type-matrix product type matrix]. The way to add the "LAN" product type is to go to the "Selection" Tab, select "Device Manager" on the left, then right click on the TAP adapter and add the feature for Device.Network.LAN.PM (power management). HLK then notices that all features for the "LAN" product type are met and it adds the "LAN" product type to the project.
    74 
    75 == Loading compatibility playlists ==
    76 
    77 Make sure to get the HLK Hardware Compatibility Playlists (on the main HLK download page), and apply the one for the correct context, e.g. HLK Version 1809 CompatPlaylist x64 Server.xml. The playlist narrows down the list of tests to the set required to get an HLK certification, removes some extra stress/failure verification type tests designed to help find driver crashes. There’s a “Load Playlist” option in the tests panel in the HLK studio app that you can use.
    78 
    7964= Setting up OpenVPN for HLK tests =
    8065
    81 == Overview ==
    82 
    83 The "Run tests" in HLK fail consistently unless the tap-windows6 adapter has an IPv6 gateway address. This can be resolved by installing an OpenVPN server and joining all HLK clients connect to it. Launching OpenVPN client on the HLK client creates an instance of tap-windows6 adapter on the system with proper IPv6 gateway address.
    84 
    85 Overview of the steps:
    86 
    87 * Install the latest OpenVPN 2.x on OpenVPN (Linux) server
    88 * Install the latest OpenVPN 2.x on the HLK clients
    89 * Install test-signed (to-be-HLK-tested) tap-windows6 driver on the HLK clients
    90 * Generate certificates and keys for OpenVPN with [https://github.com/OpenVPN/easy-rsa/ EasyRSA 3] and "openvpn --genkey"
    91 * Create and install configs for OpenVPN server and clients with embedded keys/certificates
    92 * Ensure that OpenVPN is enabled and running on server and clients
    93 * Verify OpenVPN connectivity
    94 
    95 == OpenVPN configuration files ==
    96 
    97 '''NOTE''': using p2p mode in OpenVPN would be simplest. The configuration files below do not use p2p, so please do not use them at this point.
    98 
    99 OpenVPN server configuration is fairly simple:
    100 
    101 {{{
    102 # OpenVPN server test configuration for Windows Hardware Lab Kit test server
    103 #
    104 # We need this for HLK "Run tests" which ping6 the gateway of the interface
    105 # bound to the driver being tested.
    106 #
    107 server 10.218.112.0 255.255.255.0
    108 server-ipv6 2001:db8:6666::1/64
    109 port 1194
    110 proto udp
    111 dev tun5
    112 comp-lzo
    113 persist-key
    114 persist-tun
    115 keepalive 10 120
    116 verb 4
    117 duplicate-cn
    118 max-clients 15
    119 status hlk-openvpn-status.log
    120 <ca>
    121 # CA certicate here
    122 </ca>
    123 
    124 <cert>
    125 # Server certificate here
    126 </cert>
    127 
    128 <key>
    129 # Server private key here
    130 </key>
    131 
    132 key-direction 0
    133 <tls-auth>
    134 # TLS auth key here
    135 </tls-auth>
    136 }}}
    137 
    138 OpenVPN setup on HLK clients:
    139 
    140 {{{
    141 # OpenVPN client test configuration for Windows Hardware Lab Kit test clients
    142 #
    143 # We need this for HLK "Run tests" which ping6 the gateway of the interface
    144 # bound to the driver being tested.
    145 #
    146 
    147 # Replace "x.x.x.x" with OpenVPN server's (public) IPv4 address
    148 remote x.x.x.x 1194
    149 client
    150 proto udp
    151 dev tun
    152 comp-lzo
    153 persist-key
    154 persist-tun
    155 keepalive 10 120
    156 verb 4
    157 status hlk-openvpn-status.log
    158 
    159 <ca>
    160 # CA certificate here
    161 </ca>
    162 
    163 <cert>
    164 # Client certificate here
    165 </cert>
    166 
    167 <key>
    168 # Client private key here
    169 </key>
    170 
    171 key-direction 1
    172 <tls-auth>
    173 # TLS auth key here
    174 </tls-auth>
    175 }}}
    176 
    177 == Enabling OpenVPN on server and clients ==
    178 
    179 Once OpenVPN server and clients are configured properly make sure that OpenVPN is running and automatically starts on boot. On OpenVPN server built on recent distros (e.g. Ubuntu 18.04) you'd do
    180 
    181 {{{
    182 $ systemctl enable openvpn-server@hlk
    183 $ systemctl start openvpn-server@hlk
    184 }}}
    185 
    186 On OpenVPN clients you'd do this from an administrator Powershell session:
     66The "Run tests" in HLK fail consistently unless the tap-windows6 adapter has an IPv6 gateway address. This can be resolved by a simple bridged peer to peer OpenVPN setup, where interface settings are configured statically outside of OpenVPN.
     67
     68First install the latest OpenVPN 2.x on the HLK client and support machine. Then install the same, test-signed (to-be-HLK-tested) tap-windows6 driver on the HLK clients. Then configure static IP, netmask, gateway, etc. for the TAP interface. Then generate a shared secret with "openvpn --genkey".
     69
     70The OpenVPN configuration files for HLK client and support machines can be identical except for the "remote" settings:
     71
     72{{{
     73dev tap
     74mode p2p
     75cipher AES-256-CBC
     76secret hlk.key
     77remote <remote>
     78verb 3
     79}}}
     80
     81The above setup is symmetric in the sense that neither node is a client or a server, and either one can initiate the connection. This is similar to what is described in the [https://openvpn.net/community-resources/static-key-mini-howto/ static key mini-howto].
     82
     83Once OpenVPN is configured properly make sure that OpenVPN is running and automatically starts on boot:
    18784
    18885{{{
     
    19087}}}
    19188
    192 == Peer to Peer OpenVPN setup for HLK ==
    193 
    194 The following was tested with Windows 10 x64 and worked well.
    195 
    196 What you need: 
    197 1) HLK Controller (Tested With: Windows Server 2012 R2 on VMWare and Installed HLK tools on that) 
    198 2) Two Test Machines that have similar performance characteristics (preferably exactly same builds) 
    199 3) A quality fast network switch. Slow switches can cause issues on some of the tests.
    200 
    201 Setup Steps: 
    202 1) On both test machines install the OS you want to test on. This must match the HLK tools you installed on your controller. 
    203 2) On both test machines install OpenVPN. Only the service is needed. 
    204 
    205 In the C:\Program Files\OpenVPN\config directory on each machine add configurations for a peer to peer static key configuration by following the steps here: [https://openvpn.net/community-resources/static-key-mini-howto/]. However instead of dev tun use dev tap and make the needed changes to ifconfig. Examples below: 
    206 
    207 server.ovpn
    208 {{{
    209 dev tap
    210 ifconfig 10.1.1.1 255.255.255.0
    211 secret static.key
    212 }}}
    213 
    214 client.ovpn
    215 {{{
    216 dev tap
    217 remote <IP FOR SERVER>
    218 ifconfig 10.1.1.2 255.255.255.0
    219 secret static.key
    220 }}}
    221 
    222 On each machine set the OpenVPN Legacy Service to Automatic.
    223 
    224 Reboot and ping each host to make sure everything is working. You are now setup to run the WHQL tests. Some of them need babysitting and are listed below:
    225 
    226 In general the rest of the tests passed without a problem. However, it was more successful doing groups at a time and making sure they passed.
    227 
    228 Some HLK Errata/Random Notes:
    229 * The controller seems to arbitrarily pick which machine is Support and which one is under Test. If it has trouble picking name one of the TAP adapters SupportDevice0. Picking the options in the UI didn't change the behavior. So for the tests you are baby sitting pay attention to which the server and and which one the client is. The server will run a server.htm in NDIS test so it will be obvious.
    230 
    231 * Reboots seems to happen randomly during test setup. This can be a nuisance if you are monitoring the services window or the network connections window. Helpful to make shortcuts to these so they can easily be opened.
    232 
    233 * You will have to rename your Ethernet adapters to MessageDevice. Right click on the interface and select Rename. If you attach the kernel debugger you will need to do this after attaching the debugger as it places a new interface on top of the Ethernet interface.
    234 
    235 * End to End testing can take about 5 hours because of baby sitting. Pick a good book to read or have some other work handy to keep you from losing your mind :).
     89Some have had more luck with the legacy service (OpenVPNServiceLegacy).
     90
     91Reboot and ping each host to make sure everything is working.
     92
     93
    23694
    23795Not sure if the below had any effect but was changed when doing this test:
     
    252110 * 2001:db8:6666::2
    253111
     112= Preparing HLK studio/controller =
     113
     114== Set product type ==
     115
     116In the HLK Studio's Projet tab you will see a category called "Product Types" at the right, like in [https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/hlk-studio here]. When you create a new project the "Product Types" section will be empty, and you need to make it says LAN, or you end up doing all the testing work and only get certified as an "Other Driver".
     117
     118HLK product types are listed in the [https://docs.microsoft.com/en-us/windows-hardware/test/hlk/user/hlk-product-type-matrix product type matrix]. The way to add the "LAN" product type is to go to the "Selection" Tab, select "Device Manager" on the left, then right click on the TAP adapter and add the feature for Device.Network.LAN.PM (power management). HLK then notices that all features for the "LAN" product type are met and it adds the "LAN" product type to the project.
     119
     120== Loading compatibility playlists ==
     121
     122Make sure to get the HLK Hardware Compatibility Playlists (on the main HLK download page), and apply the one for the correct context, e.g. HLK Version 1809 CompatPlaylist x64 Server.xml. The playlist narrows down the list of tests to the set required to get an HLK certification, removes some extra stress/failure verification type tests designed to help find driver crashes. There’s a “Load Playlist” option in the tests panel in the HLK studio app that you can use.
     123
     124== Adding machines to pool ==
     125
     126It seems required/useful to add HLK clients to the pool first, then the support machine. Otherwise when you select the driver in "Device manager" tab HLK will assume that the HLK support machine has the driver under test. This ''may'' create problems down the line.
     127
    254128= Configuring and running the mandatory HLK tests =
    255 
    256 == Adding machines to pool ==
    257 
    258 It seems required/useful to add HLK clients to the pool first, then the support machine. Otherwise when you select the driver in "Device manager" tab HLK will assume that the HLK support machine has the driver under test. This ''may'' create problems down the line.
    259129
    260130== Multi-machine tests ==
     
    478348The HLK client version (e.g. Windows 1809) needs to match the HLK version (HLK 1809). If there is a mismatch there can be some setup issues (some semantics of driver verifier configuration changed, or some of the test components might have failed on the OS version).
    479349
     350== Controller misuses support machine ==
     351
     352The controller seems to arbitrarily pick which machine is Support and which one is under Test. If it has trouble picking name one of the TAP adapters SupportDevice0. Picking the options in the UI didn't change the behavior. So for the tests you are baby sitting pay attention to which the server and and which one the client is. The server will run a server.htm in NDIS test so it will be obvious.
     353
     354== Reboots ==
     355
     356Reboots seems to happen randomly during test setup. This can be a nuisance if you are monitoring the services window or the network connections window. Helpful to make shortcuts to these so they can easily be opened.
     357
    480358= HLK logging =
    481359
    482 HLK logs problems mostly to event logs, so when tests are failing in an unusual fashion you can possibly find out why in the event viewer.
     360HLK controller logs problems mostly to event logs, so when tests are failing in an unusual fashion you can possibly find out why in the event viewer. Causes for test failures are generally well visible from the logs in HLK Studio.
    483361
    484362= Debugging =