Last Updated on July 22, 2022 by rudyooms
In this blog, I will be talking about a sudden “HardwareMismatchDetected” I got when I was trying to enroll my Windows 10/11 device with Autopilot for Pre-provisioning deployments.
Please Note: On 23-05-2022, exactly 1 month after experiencing the same issue I experienced the same weird behavior and it definitely looks like I am not the only one experiencing it
I will divide this blog into multiple parts
- The Issue
- Troubleshooting the missing Autopilot profile
- Solving it
- What could have caused it?
In one of my latest blogs, I showed you that with the latest Windows 11 Insider preview version, Microsoft fixed the ESP progress page after it rebooted because WUfB was targeted at devices as mentioned in this blog
As we all know, when we want to make use of the Autopilot Service we need to make sure we have the 4K Hardware Hash uploaded to Intune, otherwise, the device won’t be approved to enroll into Azure Ad and Intune.
To make sure the device was recognized I opened a new PowerShell session on my cleaned installed Dell Latitude 3590 and uploaded the Hash by using the get-windowsautopilotinfo PowerShell module. After a couple of minutes, it was done and the device ended up in the list of Autopilot devices in Intune.
Before I started my tests, I made sure the “Create Imported Windows Autopilot Device” event was logged in the Intune audit log. As shown below, on 06-04-2022 a new Autopilot device was imported!
A small reminder: When using Autopilot White Glove/Pre-Provisioning you will need to delete the Intune Object first before re-enrolling the same device
I guess when you have read the Don’t Be a Menace to Autopilot While configuring Your WUfB in the Hood blog, you can probably guess how many times I needed to remove the Intune device object (Delete MangedDevice) before testing it again.
As shown above, I tested it multiple times, that’s for sure!. But suddenly on 22-04-2022 and on 23-05-2022 the enrollment just stopped working!
2. The Issue
Just one day before I went on holiday, I heard the exciting news that with the latest Windows 11 insider preview, the Autopilot sealing issue would be fixed. Okay… I was like, I am not on vacation yet, so I deleted the Intune Object again and started the enrollment process again!
I made sure I waited 30 minutes before pressing the Windows logo to start the pre-provisioning because I also noticed some weird behavior when you try to pre-provision a device when the Intune Object was still pending deletion!
After selecting the Pre-Provisioning option as shown above, almost immediately the famous “Something went wrong” error was thrown!
As shown below, when using Windows 11, the Autopilot pre-provisioning process didn’t find an Autopilot profile? It’s telling us: No organization found and no profile found?
Even when trying to reinstall the device with Windows 10 and starting pre-provisioning shows us the famous Windows 10 red screen with the same message: No Profile found!
Of course, I was interested in what could happen when NOT choosing Autopilot pre-provisioning!
That’s bad, That’s way bad!!! A company-owned device is suddenly giving us the option to set it up for personal use! That’s weird because on 20-04-2022 it worked like expected.
To be sure I was not going crazy just 1 day before going on holiday I took my shovel out of my suitcase and started digging!.
3. Troubleshooting the missing Autopilot profile
Okay okay.. I will be more honest…I asked a colleague to get my shovel so I could start digging!
The first place to start, after checking if the device has internet should always be Intune when troubleshooting issues with Autopilot Profiles. I opened Intune and made sure if the Autopilot profile was still assigned to the device. As shown below… the profile status was still assigned as it should and with the proper Autopilot profile
You can never be certain! I needed to check it out even It should be weird that somehow this suddenly changed but ey who knows?
Now we are sure the Autopilot profile was still assigned, let’s take a look at the device itself! I opened a PowerShell session with the use of shift+f10 and opened the registry. I wanted to be sure if it was just a GUI glitch or if the Autopilot profile settings didn’t arrive at the device!
As shown above, it was missing some important information needed for Autopilot like the “CloudAssignedTenantId” and the “IsAutopilotDisabled”
Looking at the “IsAutopilotDisabled” value, it’s configured to zero. This is a good thing! let me explain! If this value was set to 1, that would mean that the device isn’t registered with Autopilot or it could also indicate that the Autopilot profile couldn’t be downloaded.
Please Note: The registry is not the only place where the Autopilot settings are stored. To be sure the downloaded autopilot profile was empty I also opened the AutopilotDDSZTDfile from the c:\windows\servicestate\wmansvc folder
As expected, as shown below the file itself didn’t have the required tenant settings in it. It only contained the DownloadDate and nothing more!
At that same time, I realized that this issue could be due to something that could have broken on Microsoft their Service side as mentioned in this blog
In a normal situation when using Autopilot, the Autopilot profile will be received from the Autopilot endpoint Zero Touch Deployment Directory Domain Services (ztd.dds.microsoft.com). This wonderful key could also be noticed in this Autopilot Settings registry key
I tried to spot differences in the network route to ztd.dds.microsoft.com when running tracert from multiple locations and other “working” devices
As shown above, on all other locations and working devices the output was almost the same so no routing issues for us this time and no DNS issues! I guess it’s time to open the event logs to take a look at the ModernDeployment-Diagnostic-Provider and Shell-Core event logs
As shown above, the ModernDeployment event log was throwing event 100 errors at me and mentioning it didn’t find the required Autopilot Registration ID! When taking a closer at that same event log, I noticed that just before it retrieved an empty profile, it was giving us the 0x80072F76 error!
0x80072F76 –> “The requested header was not found”
I guess we are done with the ModernDeployment-Diagnostics-Provider event log… for now….
High ho, high ho it’s up to the Shell-Core event log we go!! As shown below, the same kind of error is shown in the event 62407. “Unable to get device ZtdID”
I decided it was time to download fiddler to check what Fiddler has to tell us! Maybe somehow, someone placed a proxy inside this network without me knowing it?. After setting up Fiddler as shown in the blog above I restarted the Autopilot pre-provisioning flow.
Please note: You will need to run Fiddler as System (psexec) otherwise, there could be a possibility that you won’t get any errors.
Within a few seconds, Fiddler was showing us a connection to the same Autopilot Endpoint I showed you earlier. While communicating with ztd.dds.microsoft.com it also transferred/showed Microsoft its Hardware-Hash
Looks fine to me? Shall we look at the response I got while communicating with the Autopilot Endpoint?
HUH… HUH and again HUH? So somehow we got a 908 error code – HardwareMismatchDetected?
I guess I don’t need to explain what Hardware Mismatch Detected means…
When the device its Hardware Hash doesn’t match the one in Intune… it doesn’t get the Autopilot profile! But why? I didn’t change anything as shown earlier in the Intune Audit logs!
Could there be something wrong with the Hardware Hash itself? Some time ago I wrote a blog about why replacing the Motherboard will make sure the uploaded Hash no longer matches the one in your device.
When you have read the blog, you know we first need to export the Autopilot Hash. After exporting the Hardware hash I ran the oa3tool with the /validatehwhash=”hash” parameter.
Before showing you the output, let me first show you a list of attributes used by Autopilot that are found in the Hardware Hash itself.
As shown below, it’s telling us “there are invalid critical and important fields in the hardware hash“, and the error code 0xc0000201
To be sure, I also run the same command on a working device. Luckily both of them are telling me the ProductKeyID is an InvalidCriticalField. I guess when both of them are giving me the same message, it isn’t going to be an issue because one of them is working and the other one not
I asked Microsoft if it is possible for me to get the uploaded HardwareHash back so I could compare them. When using the OA3tool it’s also possible to decode that Hardware Hash. Maybe I could spot something weird in it (still waiting for a response on that question)
4. Solving it
I guess solving this weird issue is pretty simple…. but before solving it I still needed to check something else first!
To be sure I wasn’t going crazy I checked if the Serial number from my device was matching the device in Windows Autopilot Devices. As shown below, in my case it was matching the serial number on the device!
Please note: The screenshot from above is from the third time I encountered this issue!
Now we check if the serial number was still matching the one in Autopilot devices, remove the Autopilot device object and reupload the hash again!
After the device record was successfully deleted I reupload the hash again and after a couple of minutes of waiting the device object was ready to be used
I reinstalled the device and started the Autopilot pre-provisioning again and after a couple of seconds, it successfully downloaded the Autopilot profile and started the device setup!
5. What could have caused it?
I guess this part will be a work in progress because I also put in a support ticket at Microsoft about this issue. When looking at the facts, it looks like somehow the hash previously uploaded to Intune became invalid, but why?
Please Note this pure speculation, no string attached here… But looking at this Autopilot CSP Doc
You will probably notice the “HardwareMismatchRemediationData” part as shown below
This MS Doc is telling us “This string is used as input for calling Windows Autopilot Service to remediate a device if the device underwent a hardware change that affects its ability to use Windows Autopilot”
So you would assume that there is an auto-remediation service that will fix the feature HardwareMismatches when using Autopilot? So maybe because it detected multiple enrollment issues, it tried to remediate the Hardware Hash and by doing so it broke completely?
Again, this is purely speculation… but it sure does sound like a possible option for me (now)
I was 100% right, check out the Ask me Anything session. In the first few minutes, Juanita Baptiste is mentioning the Auto-Remediation!
You could probably ask me WHY I didn’t delete the device object at first, at that point I would answer… As it was working yesterday, why should I remove the hash today? So far as I know it doesn’t have a max amount of attempts it could be re-used.
Even when the solution is pretty simple I reached out to Microsoft Support to get to the bottom of this weird issue. Even when it is already solved I want to know what happened as this is NOT normal behavior and we don’t want this kind of weird behavior when enrolling devices at a large enterprise customer!!!
Please Note: For now this is the response I got:
“After discussing the case internally we found out that the issue occurred due to incorrect population of the device’s hash on the service”
That makes me worry because today was the second time I ran into this issue… that’s not good! Luckily I am experiencing the same issue again… but this time on a VM (not running pre-provisioning of course)