In this blog, I will be talking about a sudden “HardwareMismatchDetected” I got when I was trying to enroll my Windows 10/11 device with Autopilot for Pre-provisioning deployments.
1. Introduction
In one of my latest blogs, I showed you that with the latest Windows 11 Insider preview version, Microsoft fixed the ESP progress page after it rebooted because WUfB was targeted at devices, as mentioned in this blog
As we all know, when we want to make use of the Autopilot Service we need to make sure we have the 4K Hardware Hash uploaded to Intune, otherwise, the device won’t be approved to enroll into Azure Ad and Intune.
To make sure the device was recognized, I opened a new PowerShell session on my cleaned-installed Dell Latitude 3590 and uploaded the Hash by using the get-windowsautopilotinfo PowerShell module. After a couple of minutes, it was done and the device ended up in the list of Autopilot devices in Intune.
Before I started my tests, I ensured the “Create Imported Windows Autopilot Device” event was logged in the Intune audit log. As shown below, on 06-04-2022 a new Autopilot device was imported!
A small reminder: When using Autopilot White Glove/Pre-Provisioning you will need to delete the Intune Object first before re-enrolling the same device
I guess when you have read the Don’t Be a Menace to Autopilot While configuring Your WUfB in the Hood blog, you can probably guess how many times I needed to remove the Intune device object (Delete MangedDevice) before testing it again.
As shown above, I tested it multiple times, that’s for sure!. But suddenly on 22-04-2022 and on 23-05-2022 the enrollment just stopped working!
2. The Issue
Just one day before I went on holiday, I heard the exciting news that the Autopilot sealing issue would be fixed with the latest Windows 11 insider preview. Okay… I was like, I am not on vacation yet, so I deleted the Intune Object again and started the enrollment process again!
I made sure I waited 30 minutes before pressing the Windows logo to start the pre-provisioning because I also noticed some weird behavior when you try to pre-provision a device when the Intune Object was still pending deletion!
After selecting the Pre-Provisioning option as shown above, almost immediately, the famous “Something went wrong” error was thrown!
As shown below, when using Windows 11, the Autopilot pre-provisioning process didn’t find an Autopilot profile? It’s telling us: No organization found and no profile found?
Even when trying to reinstall the device with Windows 10 and starting pre-provisioning, we get the famous Windows 10 red screen with the same message: No Profile found!
Of course, I was interested in what could happen when NOT choosing Autopilot pre-provisioning!
That’s bad—that’s way bad!!! A company-owned device is suddenly giving us the option to set it up for personal use! That’s weird because on 20-04-2022, it worked like expected.
To be sure I was not going crazy, just one day before going on holiday, I took my shovel out of my suitcase and started digging!
3. Troubleshooting the missing Autopilot profile
Okay okay. I will be more honest…I asked a colleague to get my shovel so I could start digging!
The first place to start, after checking if the device has internet should always be Intune when troubleshooting issues with Autopilot Profiles. I opened Intune and made sure if the Autopilot profile was still assigned to the device. As shown below… the profile status was still assigned as it should and with the proper Autopilot profile
You can never be certain! I needed to check it out. It should be weird that somehow, this suddenly changed, but hey, who knows?
Now we are sure the Autopilot profile was still assigned, let’s take a look at the device itself! I opened a PowerShell session using shift+f10 and opened the registry. I wanted to be sure if it was just a GUI glitch or if the Autopilot profile settings didn’t arrive at the device!
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Provisioning\Diagnostics\AutoPilot
As shown above, it was missing some important information needed for Autopilot like the “CloudAssignedTenantId” and the “IsAutopilotDisabled”
Looking at the “IsAutopilotDisabled” value, it’s configured to zero. This is a good thing! let me explain! If this value was set to 1, that would mean that the device isn’t registered with Autopilot or it could also indicate that the Autopilot profile couldn’t be downloaded.
Please Note: The registry is not the only place where the Autopilot settings are stored. To be sure the downloaded autopilot profile was empty, I also opened the AutopilotDDSZTDfile from the c:\windows\servicestate\wmansvc folder
As expected, the file itself, as shown below, didn’t contain the required tenant settings. It only contained the DownloadDate!
At that same time, I realized that this issue could be due to something that could have broken on Microsoft’s Service side as mentioned in this blog.
https://www.isintunedown.com/autopilot/oobeaadv10-return-of-the-error
In a normal situation, when using Autopilot, the Autopilot profile will be received from the Autopilot endpoint Zero Touch Deployment Directory Domain Services (ztd.dds.microsoft.com). This wonderful key could also be noticed in this Autopilot Settings registry key
Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Provisioning\AutopilotSettings
I tried to spot differences in the network route to ztd.dds.microsoft.com when running tracert from multiple locations and other “working” devices
As shown above, on all other locations and working devices the output was almost the same so no routing issues for us this time and no DNS issues! I guess it’s time to open the event logs to take a look at the ModernDeployment-Diagnostic-Provider and Shell-Core event logs
As shown above, the ModernDeployment event log was throwing event 100 errors at me and mentioning it didn’t find the required Autopilot Registration ID! When taking a closer at that same event log, I noticed that just before it retrieved an empty profile, it was giving us the 0x80072F76 error!
0x80072F76 –> “The requested header was not found”
I guess we are done with the ModernDeployment-Diagnostics-Provider event log… for now….
High ho, high ho, it’s up to the Shell-Core event log we go!! As shown below, the same kind of error is shown in event 62407. “Unable to get device ZtdID”
I decided it was time to download fiddler to check what Fiddler has to tell us! Maybe somehow, someone placed a proxy inside this network without me knowing it?. After setting up Fiddler as shown in the blog above I restarted the Autopilot pre-provisioning flow.
Please note: You will need to run Fiddler as System (psexec); otherwise, you may not get any errors.
Within a few seconds, Fiddler was showing us a connection to the same Autopilot Endpoint I showed you earlier. While communicating with ztd.dds.microsoft.com it also transferred/showed Microsoft its Hardware-Hash
Looks fine to me? Shall we look at the response I got while communicating with the Autopilot Endpoint?
HUH… HUH and again HUH? So somehow, we got a 908 – HardwareMismatchDetected?
I guess I don’t need to explain what Hardware Mismatch Detected means…
When the device its Hardware Hash doesn’t match the one in Intune… it doesn’t get the Autopilot profile! But why? I didn’t change anything as shown earlier in the Intune Audit logs!
Could there be something wrong with the Hardware Hash itself? Some time ago, I wrote a blog about why replacing the Motherboard will ensure that the uploaded Hash no longer matches the one in your device.
https://call4cloud.nl/2021/12/married-with-systemboards-976-tpm
When you have read the blog, you know we first need to export the Autopilot Hash. After exporting the Hardware hash I ran the oa3tool with the /validatehwhash=”hash” parameter.
Before showing you the output, let me show you a list of attributes Autopilot used in the Hardware Hash itself.
As shown below, it’s telling us “there are invalid critical and important fields in the hardware hash“, and the error code 0xc0000201
To be sure, I also run the same command on a working device. Luckily both of them are telling me the ProductKeyID is an InvalidCriticalField. I guess when both of them are giving me the same message, it isn’t going to be an issue because one of them is working and the other one not
I asked Microsoft if it is possible for me to get the uploaded HardwareHash back so I could compare them. When using the OA3tool it’s also possible to decode that Hardware Hash. Maybe I could spot something weird in it (still waiting for a response on that question)
4. Solving it
I guess solving this weird issue is pretty simple…. but before solving it I still needed to check something else first! To be sure I wasn’t going crazy, I checked if the Serial number from my device matched the device in Windows Autopilot Devices. As shown below, it did!
Please note: The screenshot above is the third time I encountered this issue!
Now we check if the serial number still matches the one in Autopilot devices, remove the Autopilot device object, and reupload the hash again!
After the device record was successfully deleted, I reuploaded the hash again, and after a couple of minutes of waiting, the device object was ready to be used.
I reinstalled the device and started the Autopilot pre-provisioning again. After a couple of seconds, it successfully downloaded the Autopilot profile and started the device setup!
5. What could have caused it?
I guess this part will be a work in progress because I also submitted a support ticket at Microsoft about this issue. When looking at the facts, it seems that somehow, the hash previously uploaded to Intune became invalid, but why?
Please Note this is pure speculation; there no strings attached here… But looking at this Autopilot CSP Doc
WindowsAutoPilot CSP – Windows Client Management | Microsoft Docs
You will probably notice the “HardwareMismatchRemediationData” part as shown below
This MS Doc tells us “This string is used as input for calling Windows Autopilot Service to remediate a device if the device underwent a hardware change that affects its ability to use Windows Autopilot”
I guess I was right all along because after writing this blog below and looking back at this blog, it was indeed because of the HardwareMismatchRemediationData functionality!
6. Conclusions
You could probably ask me WHY I didn’t delete the device object at first. At that point, I would answer… As it was working yesterday, why should I remove the hash today? So far as I know, it doesn’t have a maximum number of attempts; it could be re-used.
Even when the solution is pretty simple I reached out to Microsoft Support to get to the bottom of this weird issue. Even when it is already solved I want to know what happened as this is NOT normal behavior and we don’t want this kind of weird behavior when enrolling devices at a large enterprise customer!!!
Please Note: For now, this is the response I got:
“After discussing the case internally, we found out that the issue occurred due to incorrect population of the device’s hash on the service”
That makes me worry because today was the second time I ran into this issue… that’s not good! Luckily I am experiencing the same issue again… but this time on a VM (not running pre-provisioning of course)
First off, thanks for this post, it was just what I was looking for.
This is interesting, because today I experienced the exact same issue. We had a bunch of new computers delivered the other day, and they are all Autopilot registered when they arrive, so we don’t have to generate and upload the hash.
I got the red screen on my first try, but on the second try it worked. But as this was a new model for us, I was doing some testing so I had to wipe it. And after that, it gave me the red screen again. I tried wiping it again, with the same result.
I will try to reupload the hash, and see if that works.
Hi…nice to hear… curious if it will fix your Autopilot issue. At this time I am again experiencing the same behavior and have an open case with MS now 🙂
Thanks for your in-depth exploration – this issue also occurred for me and although the size of our tenant is not huge, it is a worry when trying to record and validate a process for hash gathering.
Confirmed that after removal of the enrolled device from AP, re-exporting the hash from the test device and then reimporting into AP, the AP profile was correctly downloaded and processed.
Thanks for the post. I was able to BYPASS AutoPilot on a laptop from a company that’s closing and allowing the employees to keep the systems using some of the information you provided. Interesting how autopilot registers itself even after completely wiping / reloading the OS drive. I’ll need to look into that more. In the meantime, keep up the great writing!
You are my hero!
Thank you so much for this awesome and in depth post!
Where can i read more from you?
Best regards,
Moha
Just ran into this same exact issue (thank you for the detailed deep dive). What’s even more strange, As an MS Partner, we can upload devices to AP through the Partner Portal using the tuple (Make/Model/Serial) and we were still bitten by this. After deleting the device from Intune/AP/AAD the issue remained. Something tells me AP keeps a hardware hash when the OEM reports the device to MS as being built. So even though we uploaded the make/model/serial it still tried to match the hardware hash.
I just experienced the same problem with one laptop in our environment.
We have a SCCM task sequence that collect the hardware hash using OA3TOOL.exe and this has been working fine for many devices however in this one cases the device couldn’t find a profile when we tried to white glove it. And deleting it from Intune and running the SCCM task sequence again to reimport the hardware hash didn’t solve the problem.
I finally ended up launching the command prompt from OOBE and ran the Get-WindowsAutopilotInfo.ps1 script found here
https://learn.microsoft.com/en-us/autopilot/add-devices and the hardware hash I got by doing that didn’t match the one that OA3TOOL.exe generated.
So I deleted the device from Intune again and imported the new hardware hash and now it get’s an Autopilot profile.
Its good to know that every single time you take a look at the hash its different.. (time stamps)