Last Updated on May 17, 2023 by rudyooms

In this blog, I am going to take a good look at the Autopilot Hardware Mismatch Remediation Data. Please note! I am going to do a lot of assumptions and a bit of guessing here by looking at the AutoPilot.dll to determine how it works.

1. Introduction
2. What happens
3. The Flow
4. The Scheduled Task
5. WNF
6. When you break the remediation flow
7. Some Tips!

1. Introduction

After digging into the Autopilot profile issue that was caused by the Autopilot Marker, I decided to also dig a bit into the Hash Mismatch Remediation CSP. I will try to explain what happens behind the curtains and why the new Hardware Mismatch remediation relies on the Autopilot Marker I mentioned in the blog I mentioned above.

Before showing you what happens, it could be smart to first know what I am talking about, so let me explain a bit. With the July 26 preview release, Microsoft added the HardwareMisMatchRemedation option

WindowsAutopilot CSP – Windows Client Management | Microsoft Learn

When looking at the little official Microsoft documentation available, Microsoft is telling us that this Windows Autopilot CSP collects hardware information about a device that is used when the device needs remediation if it underwent a hardware change

If an Autopilot device underwent a Hardware Change we will notice these 2 messages in the Device its profile status. The first one is mentioning the fact there is a “Fix Pending”

When we take a closer look at the Profile Status itself it mentioned that a hardware change was detected on this device.

Shall we take a closer look at what I guess happens when the device underwent a hardware change?

2. What Happens

Before I did anything else, I manually run the DetectHardwareChange scheduled task even when there were no changes to make sure the HardwareInfo was available in the registry, so I can be sure the Autopilot service has something to compare.

Please note: I will try to explain more about this scheduled task in part 4 of this blog

This DetectHardwareChange scheduled task creates a “snapshot” with the hardware specifications of the device, some stuff is stored in the registry, and some of it is on the disk. Let’s start by looking at the registry, as shown below, the provisioning\hardwareinfo registry key is the one to look at.

As shown below, this registry key stores some important information about the device itself. I guess the AutopilotMarker sounds familiar?

Besides the registry, we could also spot some disk activity with the use of Procmon. We will notice that a cachedhash.txt and the cachedtpmekpub.txt will be placed on the disk

You could ask yourself the question: Why are those files stored in the wmansvc folder AKA Windows Management Service (Windows.Management.Service.Dll)….because it’s the same location as in which the Autopilot JSON is stored, so I guess it was a well-known place to store these TXT files

After making sure that the scheduled task was triggered once, I shutdown the VM and “changed” some important stuff like the UUID and the BIOSSerialNumber as shown below

Script: Change the BIOS GUID of a Hyper-V Virtual Machine (altaro.com)

After changing those Hardware Identifiers I booted the VM and determined if the serial was indeed changed

As shown above, the BiosSerialnumber was indeed changed. Now I am pretty sure the Hardware Identifiers changed I started Fiddler and opened the scheduled tasks again to launch the “DetectHardwareChange” scheduled task

After launching that task (which is normally automatically launched because of a “hardware event” detected by WNF), within a few seconds, some additional files with LastReported popped up in the wmansvc folder

We can also spot the same behavior in the HardwareInfo registry key I mentioned earlier

After those keys arrived at the registry it was time to open the event log. Look at what I got! A nice ModernDeployment-Diagnostics-Provider event 122 mentioning the “Autopilot task” to detect hardware change finished successfully: a change was detected and reported. So my guess is it compares the “Orginal/LastReported” registry keys and the new ones

At the same time that event pops up, we notice that Windows is also trying to reach out to the Autopilot Service ztd.dds.microsoft.com to Patch the UpdateDeviceHardwareMismatchRemediationState

Please Note: This Autopilot URL could also be spotted when looking at the Autopilot.Dll file as shown below

When looking at the headers in the PATCH request to ztd.microsoft.com we will notice that it will contain the “HardwareMismatchRemediationData”

When decompiling the “HardwareMismatchRemediationData” we will notice that it contains the CachedHardwareBlob and the CurrentHardwareBlob. Let’s start with the Cashed one! As shown below this CachedHardwareBlob has the same contents as the cachedhash.txt we noticed in the wmansvc folder

When looking at the second blob AKA the CachedHardwareBlob and decoding it with the OA3TOOL. We will notice that this blob is just the old hardware with the “OLD” serial and UUID in it.

When looking back at the CurrentHardwareBlob and doing the same as with the Cached one we will notice that this one has the NEW Serial number and the new UUID .

This information is generated by using the ./DevDetail/Ext/DeviceHardwareData CSP

Besides this information, we will also notice some information from the Autopilot Profile. For example, the Autopilot marker I blogged about last week

Of course, this takes place in the system context but if you want to know which MSA tickets are cached for your user, you could take a look at the Software\Microsoft\IdentityCRL\Immersive\production\Token registry key.

If you want to know more about this registry location, you could take a look at my previous blog. With all of this information sent to the Autopilot service (CurrentHardwareBlob, CachedHardwareBlob, AutopilotMarker, IsUEFISupported) Microsoft will look at the details about the hardware change it got and will try to “fix” it. When looking at other Official documentation I assume Microsoft will look for this information in the BLOBs

Let’s continue and let’s take a look when we open the Windows Autopilot devices blade and press the sync button

We will notice that the device with the old serial number mentions that a hardware change was detected on this device and they are trying to automatically register the new hardware!!

After 3 days of waiting the Profile Status was still showing “Fix Pending”

If this is the way it works…and we need to wait such a long time before it is fixed I rather remove the hash and reupload it again. To be sure it wasn’t already fixed in the background I reinstalled the same VM and run the same PowerShell scripts I mentioned in my previous blog.

As shown above, after waiting 3 days the device is still showing: ZtdDeviceIsNotRegistered. So I guess I need to wait a bit longer?

3. The Flow

4. Detect Hardware Change Scheduled Task

I stumbled upon this scheduled task by looking at the WPR trace I got. As shown below the trace is showing us the DetectHardwareChangeHandler and it is mentioning “EnableSpecificTask” and the corresponding task “RemediateHardwareChange”

After exporting all scheduled tasks with PowerShell it showed me where to look. It showed me the \Microsoft\Windows\Management\Autopilot scheduled task location

Of course, the first thing I did was open Procmon and launch the DetectHardwareChange scheduled task. As shown below svchost.exe is going through all the hardware and the TPM. I also noticed the CLIPC.Dll being called upon, so I assume it also performs some licensing checks in the background!

I guess that task really goes through all the hardware to detect if there were any changes made

The next thing I did was export the task to XML with the use of the schtasks executable, just like I did when I needed to export the TPM-Maintenance task

Ready For Attestation False and the 0x81039001 TPM time-out (call4cloud.nl)

As shown above we will notice the WnfStateChangeTrigger and the Statename: 7540bca3201d8741. When trying to find anything useful on google about this statename, you will be out of luck.

Shall I let you into a little secret? We need to read it backward, just like I did when I was playing around to retrieve the tenant Id from the Intune MDM certificate

function Translate-StateName {
    param (
        [ValidateLength(16,16)] [String] $StateName
    )
    
    process {
        $CharArray = (&{for ($i = 0;$i -lt $StateName.length;$i += 2) { $StateName.substring($i,2) }})
        $TranslatedStateName = (&{for ( $i = 7; $i -ge 0; $i-- ) { $CharArray[$i] }}) -join ''
        return $TranslatedStateName
    }
}

This PowerShell script from above will give you the option to translate that WNF statename to something more useful.

This TaskName: 41871D20A3BC3475 will show you one hit on google but I decided to open IDA again and open the ContentDeliveryManager.utilities.dll

Guess why I opened this specific DLL! Because that’s the DLL with all the Windows Notification Facility IDs in it! When searching for this specific ID, it showed me a couple of hits

As shown above and below: It’s again mentioning the: WNF_MSA_HARDWARE_DEVICE_IDENTITY_UPDATED

After spending some time on google I stumbled upon this WNF trigger: WNF_MSA_ACCOUNTSTATECHANGE

That one led me to the Wlidsvc.dll AKA Windows Live ID. Windows Live ID… that one rings a bell or two when combining it with MSA?

Again and again each time it is mentioning WNF. I guess I need to take a look and explain a bit about what WNF is

5. WNF

WNF stands for Windows Notification Facility… Whoop part 5 is finished.

Not yet I guess because as I was mentioning in the first part, I manually triggered that Scheduled Task. As mentioned before the scheduled task trigger points to WNF, so let me try to explain how this is triggered.

WNF is the notification system within the Windows Operating System. As we noticed in part 4, the task scheduler can subscribe to an event before it’s even occurring and it could launch a scheduled task against it.

So to put it in simple words… When every time a hardware change event occurs it will launch that Detect Hardware Change scheduled task.

Let me give you a simple example: If you want to be notified each time an application is launched from the desktop we could create a scheduled task and define the StateChangeTrigger: 7550BEA33E06830D AKA WNF_SHEL_DESKTOP_APPLICATION_STARTED

When adding an action to this scheduled task, we can make sure a specific action will be triggered when this type of event occurs. This action could trigger a DLL function or just a simple PowerShell script that outputs a file to a directory each time a desktop application is launched

I guess I love WNF… because without this Trigger the DetectHardwareChange scheduled task isn’t going to be executed and without it… no Hardware Remediation

6. When you break the remediation Flow

I guess you know me…. If you don’t break it, you never know how you could put it back together. So I did the same with the Hash Remediation. I enrolled a new VM with a vTPM uploaded the hash and enrolled the device with Autopilot.

I disabled Bitlocker and shut down the VM. I created a new VM and attached that hard disk and booted the device. After running the dsregcmd /forcerecovery to fix the broken TPM authentication, I decided to run the DetectHardwareChange scheduled task.

After a few seconds, the device-management-diagnostics\autopilot event log was filled with the error… As shown below it’s mentioning an invalid remediation request and that the device profile cannot be null or empty.

Besides this error, it also showed me the fact that the Autopilot hardware remediation report failed with the error: 0x80190190. I have a slight feeling that the Autopilot hash remediation isn’t going to work.

Of course, I also enabled the Fiddler trace to determine if I could spot something different going on

7. Some Tips

To finish off this blog post, here are some tips.

1. Don’t rebuild your device manually! Use the “Wipe” option in Intune! Otherwise, you could be stuck with an older OS build that doesn’t know about the Autopilot Marker. If you rebuild it manually, please make sure you do so with the latest Windows Build!!!
2. If a hardware change occurred (motherboard replacement) this is indeed the message you could receive. If you are in a hurry delete the Autopilot Object (hash..which is not an hash) and upload it again
3. I am hearing people mentioning the fact, that when the bios/firmware was updated they also could get the same message. If that’s the case please leave me a message if you also put in a Microsoft Ticket!

Conclusion

Stumbling upon the Autopilot Profile download issue, let me do some digging and some more digging. Even when there is still no Official Microsoft documentation about the Autopilot Hash Remediation, I hope this blog showed you a bit about how it could possibly work