The Incredibly Strange Device Who Stopped Syncing and Became Certificate Zombies!!?

This blog will NOT be about the new Autopatch function! Of course not! This blog will be about multiple weird Intune Device sync errors I was asked to troubleshoot.

I will try to explain everything I looked into and how I started troubleshooting to make sure it was fixed when I was done!

1. Introduction

I am going to mention 2 separate issues. One was on Reddit and the other was on the MEM group on LinkedIn. Both of them shared the same issue and error codes but still, they were 2 separate issues

1. Intune: Mobile Device Management (reddit.com)

Looking at the first issue he mentions that several computers have “dropped out of Intune”

Okay… that’s something you don’t see every day? Let’s look at the second question/issue I got on the MEM group on Linkedin

2. Modern Endpoint Management (MECM | SCCM | Intune | AzureVD | Security | MacOS | iOS ) | Groepen | LinkedIn

Looking at the picture below he is mentioning OMA-DM message failed to be sent. Result: (Unknown Win32 Error code: 0x80072f99

2. Troubleshooting command

When you need to perform some remote troubleshooting, you could always run the remote diagnostics in Intune but this time I asked them to run a nice PowerShell command on the device itself

wget https://aka.ms/intuneps1 -outfile IntuneODCStandAlone.ps1

powerShell -ExecutionPolicy Bypass -File .\IntuneODCStandAlone.ps1

this wonderful command downloads the required XML file from Github

https://raw.githubusercontent.com/markstan/IntuneOneDataCollector/master/Intune.xml

This XML contains all the locations the PowerShell script needs to pull data from. It’s a need to have when you need to perform some basic troubleshooting

After running this PowerShell command, you will end up with a nice folder with a lot of information in it!! And I mean a lot!

After receiving the zip file with all the diagnostic information, I decided to check out the error code I got first, as it could tell me in which corner I needed to start digging.

3. 0x80072f99

When troubleshooting, an error code is always helpful. If you have an error code and don’t know what it means, open the wonderful CMTrace tool and perform the Error Lookup.

As shown below, I looked up the error code 0x80072f99.

Let me translate it if your Dutch isn’t good enough to read it. No credentials were available in the client certificate AKA ERROR_WINHTTP_CLIENT_CERT_NO_PRIVATE_KEY.

The client certificate? That needs to be the Intune device MDM certificate… no question about it! Luckily I already wrote a blog about sync issues and the error code 0x80190190 some time ago. So please read it to get a better understanding of the Intune MDM device certificate

4. Troubleshooting MDM Certificate Issues

Before we troubleshoot, it’s always best practice to ensure there isn’t anything wrong in the DSREGCMD /status output. Luckily that file is also included in the zip file

It does look fine, does it? Let’s go back to the 0x80072f99 we got! I will guide you through the troubleshooting steps I always take to find the root cause of an issue.

When dealing with Intune Device certificate issues the first thing I always do is check if the GUID matches the enrollment of the Intune Management Extension (IME) log

As shown above, it mentions the same enrollment as I noticed in one of the files I got: Microsoft_Enrollments.txt

With this content software\microsoft\enrollments in it!

When looking at the text file and the enrollment information we notice it’s mentioning the DMPCertThumbPrint. You need to be sure if that one matches the information in the Set MdmDeviceCertificate part in the IME LOG

As shown above, this time it matches the right certificate.. No issues here, right?

Most of the time when you have sync errors, you are also experiencing issues with the Company Portal. So I decided to also take a look at the log file of the Company Portal! ….

When running this PowerShell script, it will fetch this log from the AppData\Local\Packages\Microsoft.CompanyPortal_8wekyb3d8bbwe\LocalState folder as shown below

After opening this Company Portal log file, I immediately noticed the log is telling me when evaluating if the MDM certificates has a matching local device, there was no matching local device found

P.S: This log also contains information about whether your device is compliant. Combining this log and first running this command intuneintunemanagementextension://synccompliance could give you the information you need to show users a toast message?

Until now, I didn’t have opened the event log once! Shall we open some of them? just for good old sakes? So I guess it’s time to take a good look at the certificates and the corresponding event log. I really hoped there was something mentioned in the Crypto-NCrypt event log. This event log is also included in the zip file, isn’t that just wonderful?

As shown above, a lot of errors, so there is something wrong… that’s for sure. It is only showing us the error 0x80090029

That specific error only means that “The requested operation is not supported”. AKA (-2146893783 NTE_NOT_SUPPORTED)

Okay, let’s close the event log as it isn’t telling us anything useful! But luckily we have enough information left from the diagnostic zip file because when running the troubleshooting command, it is also exporting the certificate part of the registry

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\MY\Certificates\

This export will be placed inside Microsoft_SystemCertificates.txt . I pulled out that part and created a reg file from it and just imported them

As shown above, this is my own test device… so it also shows my own Microsoft Intune MDM device Ca (49fc1700-x-x-x-x). So please ignore that one. You will notice it has 2 Microsoft Intune Device Certs! Huh did I miss something? I reopened the IME log. And yes as shown below, it’s also mentioning it finds 2 MDM certificates

5. Dealing with lingering Expired MDM Certificates

Having 2 MDM certificates is not the way it should be, because when you have lingering Intune MDM device certificates you could run into the same issue as I was telling you in the introduction

As shown above, when you run into the 0x80072f99 sync error, make sure you only have 1 valid MDM certificate. If one of them is expired, remove that one NOT the still valid one!! Now let’s continue to solve the second issue because by removing the duplicate certificate our second issue was still not resolved.

6. Something is Missing

Shall we take another look at the certificates I got? Because looking at certificates is always fun!

As shown above, it looks a lot like it is missing an icon which the expired one does have

This icon above will normally show you if you have the private key for this certificate. It’s even mentioned in the System Event log. Normally, I am good at ignoring Schannel events, but this one (36869) also mentions that the certificate doesn’t have the private key information property attached to it!

The System Event log is also a good place to start looking when you have Certificate issues. So I asked him to run this command to find out if the certificate key pair was still on the device.

$a = Get-ChildItem Cert:\LocalMachine\My\ | where{$_.issuer -like "*Microsoft Intune MDM Device*"}
$a = $a.thumbprint
$a = Get-Item Cert:\LocalMachine\My\$a
$a.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName

As shown above, the output of the PowerShell script is mentioning a “Unique Container Name”. When opening the c:\programdata\microsoft\crypto\keys there needs to be a file with that exact same name! Luckily in this case the key pair still was still available on the device, so what is going on here?

7. Certutil

Back in the good old days, I also stumbled upon this issue when importing a certificate on a server. After importing it was immediately removed. Back then I solved it by running the command certutil -repairstore my “Thumbprint of the certificate”

I decided to automate it a little bit

Write-Host "List certificates without private key: " -NoNewline
$certsWithoutKey = Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.HasPrivateKey -eq $false}
if($certsWithoutKey) {
    Write-Host "V" -ForegroundColor Green
    $Choice = $certsWithoutKey | Select-Object Subject, Issuer, NotAfter, ThumbPrint | Out-Gridview -Passthru
    if($Choice){
        Write-Host "Search private key for $($Choice.Thumbprint): " -NoNewline
        $Output = certutil -repairstore my "$($Choice.Thumbprint)"
        $Result = [regex]::match($output, "CertUtil: (.*)").Groups[1].Value
        if($Result -eq '-repairstore command completed successfully.') {
            Write-Host "V" -ForegroundColor Green
        } else {
            Write-Host $Result -ForegroundColor Red
        }
    } else {
        Write-Host "No choice was made." -ForegroundColor DarkYellow
    }
} else {
    Write-Host "There were no certificates found without private key." -ForegroundColor DarkYellow
}

8. Combining all the steps

To solve the second issue that had a lingering MDM certificate AND a valid one but without the private key we took these 5 steps

• Deleted the expired certificate
• Ran the Certutil script, chose the certificate without exportable key
• After running the script the Certificate was fixed and the private key was paired again
• Reboot
• Restart Log on with username/ password and NOT by using Windows Hello

Another possibility would be to install this PowerShell module and run the debug tool to determine what is going on

Intune Sync Debug PowerShell Tool to fix Intune Sync Issues (call4cloud.nl)Intune Sync Debug PowerShell Tool to fix Intune Sync Issues (call4cloud.nl)

Conclusion

Certificates and Intune go together like a horse and carriage, and knowing how to troubleshoot these kinds of sync errors is always valuable information.