Kernel debugging for Windows instances over the network

The KDNET Extensibility Module for Elastic Network Adapter (ENA) is a hardware driver support layer that enables Windows kernel debugging over the network through ENA on Amazon Elastic Compute Cloud instances. You can use the extensibility module with the Windows Debugger (WinDbg) to perform kernel-level debugging on EC2 instances running Windows.

Kernel debugging helps you diagnose and troubleshoot low-level operating system issues such as blue screen errors (BSODs), driver failures, and boot problems on your EC2 Windows instances.

Prerequisites

Before you begin, make sure you have the following:

Two EC2 Windows instances, in the same subnet:

For more information about launching instances, see Get started with Amazon EC2.

Security groups for the host and target instances must allow inbound and outbound UDP traffic on the port used for KDNET debugging (recommended range: 50000–50039). The simplest way to set this up is to create a security group with an inbound rule that allows UDP traffic from itself as the source, and then attach that security group to both instances. For more information, see Security group rules in the Amazon VPC User Guide.

The debug target instance must run one of these Windows versions (or later):

Test-Path C:\Windows\system32\kd_02_1d0f.dll

Step 1: Install Windows Debugging Tools on the debug host

Install the Windows Debugging Tools on the debug host instance by running the following command in a PowerShell session:

winget install microsoft.windbg

For detailed installation instructions, see Install the Windows Debugger in the Microsoft documentation.

After installation, verify that the debugger is working by running the following command in a PowerShell session:

windbgx

The WinDbg window should open. If it does, the installation was successful and you can close the window.

Step 2: Set up the debug target

On the debug target, open an elevated PowerShell session and configure kernel debugging using the following steps.

Run the following command to list the bus, device, and function number of the ENA adapter attached to the instance:

Get-NetAdapter -Physical |
    Where-Object -Property PnPDeviceID -Match -Value '^PCI\\VEN_1D0F&DEV_EC2[01]&' |
    Get-NetAdapterHardwareInfo |
    Select-Object InterfaceDescription, BusNumber, DeviceNumber, FunctionNumber |
    Format-List

Get-NetAdapter -Physical |
    Where-Object PnPDeviceID -Match '^PCI\\VEN_1D0F&DEV_EC2[01]&' |
    ForEach-Object {
        $adapter = $_
        $hwInfo = $adapter | Get-NetAdapterHardwareInfo
        $ipInfo = Get-NetIPAddress -InterfaceIndex $adapter.InterfaceIndex -AddressFamily IPv4
        [PSCustomObject]@{
            InterfaceDescription = $adapter.InterfaceDescription
            IPAddress            = $ipInfo.IPAddress
            BusNumber            = $hwInfo.BusNumber
            DeviceNumber         = $hwInfo.DeviceNumber
            FunctionNumber       = $hwInfo.FunctionNumber
        }
    } | Format-List

Note the BusNumber, DeviceNumber, and FunctionNumber values of the ENA adapter to be used for debugging from the output.

Run the following commands to configure kernel debugging. Replace the placeholder values with your specific configuration:

bcdedit /debug on
bcdedit /set loadoptions FORCEHVTONOTSHAREDEBUGDEVICE
bcdedit /dbgsettings net hostip:host-private-ip port:port-number key:encryption-key busparams:b.d.f

(bcdedit /enum) -match "loadoptions"

Where:

bcdedit /bootdebug on

Step 3: Start the debugging session on the debug host

To allow debugging traffic on the host, you can create a firewall rule for either the WinDbg application or a specific UDP port.

Option 1: Allow the WinDbg application

Run the following commands to authorize the WinDbg executable:

$WinDbgxPath = "$env:LocalAppData\Microsoft\WindowsApps\WinDbgX.exe"
New-NetFirewallRule -DisplayName "Allow Inbound KDNET Connection" -Action Allow -Program $WinDbgxPath

Option 2: Allow a specific UDP port

Alternatively, you can allow inbound UDP traffic to the port configured for kernel debugging. Replace port-number with your chosen KDNET port:

$DebugPort = port-number
New-NetFirewallRule -DisplayName "Allow Inbound KDNET Connection" -Direction Inbound -LocalPort $DebugPort -Protocol UDP -Action Allow

Start WinDbg with the port and key that match the debug target configuration. You can specify additional kernel debugging options documented in the Microsoft WinDbg command line reference to suit your use case.

windbgx -k net:port=port-number,key=encryption-key

WinDbg opens and waits for the debug target to connect.

Step 4: Reboot the debug target

On the debug target, restart the instance to initiate the debugging connection:

shutdown -r -t 0

After the debug target restarts, WinDbg on the debug host connects to the target automatically. You can now use WinDbg to inspect the kernel state, set breakpoints, and diagnose issues.

Clean up debug settings after debugging

On the debug target

After you finish debugging, remove the kernel debugging configuration from the debug target to restore normal boot behavior. On the debug target, open an elevated PowerShell session and run the following commands:

bcdedit /debug off
bcdedit /dbgsettings LOCAL
bcdedit /deletevalue loadoptions

If you have existing loadoptions other than FORCEHVTONOTSHAREDEBUGDEVICE, you should restore the setting by running bcdedit /set loadoptions with the original loadoptions.

If you enabled boot debugging, also run:

bcdedit /bootdebug off

Restart the instance for the changes to take effect.

On the debug host

Remove the firewall rule by running:

Remove-NetFirewallRule -DisplayName "Allow Inbound KDNET Connection"

Limitations

The KDNET extensibility module for ENA does not currently support:

The module does not currently support instances with Secure Boot enabled. You can verify the status by running Confirm-SecureBootUEFI in an elevated PowerShell session. If the output is True, Secure Boot is active. Note that all Amazon-provided images with a 'TPM' prefix have Secure Boot enabled by default.

Additional notes

If you encounter issues connecting the debugger to the target instance, verify the following:

For additional guidance, see Set up KDNET network kernel debugging manually in the Microsoft documentation.