Call4Cloud | MMP-C | Autopilot | Device Preparation

The Incredibly Strange Device Who Stopped Syncing and Became Certificate Zombies!!?

Patch My Pc | install & update thousands of apps

This blog will NOT be about the new Autopatch function! Of course not! This blog will be about multiple weird Intune Device sync errors I was asked to troubleshoot.

I will try to explain everything I looked into and how I started troubleshooting to make sure it was fixed when I was done!

1. Introduction

I am going to mention 2 separate issues. One was on Reddit and the other was on the MEM group on LinkedIn. Both of them shared the same issue and error codes but still, they were 2 separate issues

1. Intune: Mobile Device Management (reddit.com)

Looking at the first issue he mentions that several computers have “dropped out of Intune”

Text  Description automatically generated

Okay… that’s something you don’t see every day? Let’s look at the second question/issue I got on the MEM group on Linkedin

2. Modern Endpoint Management (MECM | SCCM | Intune | AzureVD | Security | MacOS | iOS ) | Groepen | LinkedIn

Looking at the picture below he is mentioning OMA-DM message failed to be sent. Result: (Unknown Win32 Error code: 0x80072f99

Graphical user interface, text, application, email  Description automatically generated

2. Troubleshooting command

When you need to perform some remote troubleshooting, you could always run the remote diagnostics in Intune but this time I asked them to run a nice PowerShell command on the device itself

wget https://aka.ms/intuneps1 -outfile IntuneODCStandAlone.ps1

powerShell -ExecutionPolicy Bypass -File .\IntuneODCStandAlone.ps1

this wonderful command downloads the required XML file from Github

https://raw.githubusercontent.com/markstan/IntuneOneDataCollector/master/Intune.xml

This XML contains all the locations the PowerShell script needs to pull data from. It’s a need to have when you need to perform some basic troubleshooting

After running this PowerShell command, you will end up with a nice folder with a lot of information in it!! And I mean a lot!

Graphical user interface, text, application, email  Description automatically generated

After receiving the zip file with all the diagnostic information, I decided to check out the error code I got first, as it could tell me in which corner I needed to start digging.

3. 0x80072f99

When troubleshooting, an error code is always helpful. If you have an error code and don’t know what it means, open the wonderful CMTrace tool and perform the Error Lookup.

As shown below, I looked up the error code 0x80072f99.

checking the Intune sync error: 0x80072f99 in the cmtrace. It shows us that there No credentials were available in the client certificate AKA ERROR_WINHTTP_CLIENT_CERT_NO_PRIVATE_KEY.

Let me translate it if your Dutch isn’t good enough to read it. No credentials were available in the client certificate AKA ERROR_WINHTTP_CLIENT_CERT_NO_PRIVATE_KEY.

The client certificate? That needs to be the Intune device MDM certificate… no question about it! Luckily I already wrote a blog about sync issues and the error code 0x80190190 some time ago. So please read it to get a better understanding of the Intune MDM device certificate

4. Troubleshooting MDM Certificate Issues

Before we troubleshoot, it’s always best practice to ensure there isn’t anything wrong in the DSREGCMD /status output. Luckily that file is also included in the zip file

It does look fine, does it? Let’s go back to the 0x80072f99 we got! I will guide you through the troubleshooting steps I always take to find the root cause of an issue.

When dealing with Intune Device certificate issues the first thing I always do is check if the GUID matches the enrollment of the Intune Management Extension (IME) log

As shown above, it mentions the same enrollment as I noticed in one of the files I got: Microsoft_Enrollments.txt

With this content software\microsoft\enrollments in it!

When looking at the text file and the enrollment information we notice it’s mentioning the DMPCertThumbPrint. You need to be sure if that one matches the information in the Set MdmDeviceCertificate part in the IME LOG

Graphical user interface, text, application  Description automatically generated

As shown above, this time it matches the right certificate.. No issues here, right?

Most of the time when you have sync errors, you are also experiencing issues with the Company Portal. So I decided to also take a look at the log file of the Company Portal! ….

When running this PowerShell script, it will fetch this log from the AppData\Local\Packages\Microsoft.CompanyPortal_8wekyb3d8bbwe\LocalState folder as shown below

Graphical user interface, text, application, email  Description automatically generated

After opening this Company Portal log file, I immediately noticed the log is telling me when evaluating if the MDM certificates has a matching local device, there was no matching local device found

Graphical user interface, text, application, email  Description automatically generated

P.S: This log also contains information about whether your device is compliant. Combining this log and first running this command intuneintunemanagementextension://synccompliance could give you the information you need to show users a toast message?

Until now, I didn’t have opened the event log once! Shall we open some of them? just for good old sakes? So I guess it’s time to take a good look at the certificates and the corresponding event log. I really hoped there was something mentioned in the Crypto-NCrypt event log. This event log is also included in the zip file, isn’t that just wonderful?

 0x80090029
2146893783 NTE_NOT_SUPPORTED

As shown above, a lot of errors, so there is something wrong… that’s for sure. It is only showing us the error 0x80090029

That specific error only means that “The requested operation is not supported”. AKA (-2146893783 NTE_NOT_SUPPORTED)

Okay, let’s close the event log as it isn’t telling us anything useful! But luckily we have enough information left from the diagnostic zip file because when running the troubleshooting command, it is also exporting the certificate part of the registry

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\SystemCertificates\MY\Certificates\

This export will be placed inside Microsoft_SystemCertificates.txt . I pulled out that part and created a reg file from it and just imported them Graphical user interface, text, application  Description automatically generated

As shown above, this is my own test device… so it also shows my own Microsoft Intune MDM device Ca (49fc1700-x-x-x-x). So please ignore that one. You will notice it has 2 Microsoft Intune Device Certs! Huh did I miss something? I reopened the IME log. And yes as shown below, it’s also mentioning it finds 2 MDM certificates

Graphical user interface, text, application  Description automatically generated

5. Dealing with lingering Expired MDM Certificates

Having 2 MDM certificates is not the way it should be, because when you have lingering Intune MDM device certificates you could run into the same issue as I was telling you in the introduction

Graphical user interface, text  Description automatically generated with medium confidence

As shown above, when you run into the 0x80072f99 sync error, make sure you only have 1 valid MDM certificate. If one of them is expired, remove that one NOT the still valid one!! Now let’s continue to solve the second issue because by removing the duplicate certificate our second issue was still not resolved.

6. Something is Missing

Shall we take another look at the certificates I got? Because looking at certificates is always fun!

As shown above, it looks a lot like it is missing an icon which the expired one does have

This icon above will normally show you if you have the private key for this certificate. It’s even mentioned in the System Event log. Normally, I am good at ignoring Schannel events, but this one (36869) also mentions that the certificate doesn’t have the private key information property attached to it!

The System Event log is also a good place to start looking when you have Certificate issues. So I asked him to run this command to find out if the certificate key pair was still on the device.

$a = Get-ChildItem Cert:\LocalMachine\My\ | where{$_.issuer -like "*Microsoft Intune MDM Device*"}
$a = $a.thumbprint
$a = Get-Item Cert:\LocalMachine\My\$a
$a.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName
Graphical user interface, text  Description automatically generated

As shown above, the output of the PowerShell script is mentioning a “Unique Container Name”. When opening the c:\programdata\microsoft\crypto\keys there needs to be a file with that exact same name! Luckily in this case the key pair still was still available on the device, so what is going on here?

7. Certutil

Back in the good old days, I also stumbled upon this issue when importing a certificate on a server. After importing it was immediately removed. Back then I solved it by running the command certutil -repairstore my “Thumbprint of the certificate”

I decided to automate it a little bit

Write-Host "List certificates without private key: " -NoNewline
$certsWithoutKey = Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.HasPrivateKey -eq $false}
if($certsWithoutKey) {
    Write-Host "V" -ForegroundColor Green
    $Choice = $certsWithoutKey | Select-Object Subject, Issuer, NotAfter, ThumbPrint | Out-Gridview -Passthru
    if($Choice){
        Write-Host "Search private key for $($Choice.Thumbprint): " -NoNewline
        $Output = certutil -repairstore my "$($Choice.Thumbprint)"
        $Result = [regex]::match($output, "CertUtil: (.*)").Groups[1].Value
        if($Result -eq '-repairstore command completed successfully.') {
            Write-Host "V" -ForegroundColor Green
        } else {
            Write-Host $Result -ForegroundColor Red
        }
    } else {
        Write-Host "No choice was made." -ForegroundColor DarkYellow
    }
} else {
    Write-Host "There were no certificates found without private key." -ForegroundColor DarkYellow
}

8. Combining all the steps

To solve the second issue that had a lingering MDM certificate AND a valid one but without the private key we took these 5 steps

  • Deleted the expired certificate
  • Ran the Certutil script, chose the certificate without exportable key
  • After running the script the Certificate was fixed and the private key was paired again
  • Reboot
  • Restart Log on with username/ password and NOT by using Windows Hello

Another possibility would be to install this PowerShell module and run the debug tool to determine what is going on

Intune Sync Debug PowerShell Tool to fix Intune Sync Issues (call4cloud.nl)Intune Sync Debug PowerShell Tool to fix Intune Sync Issues (call4cloud.nl)

Conclusion

Certificates and Intune go together like a horse and carriage, and knowing how to troubleshoot these kinds of sync errors is always valuable information.

Its Important GIFs | Tenor

6 thoughts on “The Incredibly Strange Device Who Stopped Syncing and Became Certificate Zombies!!?

  1. Hello Rudy,

    Thank you for this helpful article.
    The only problem I encountered, I was unable to get the unique container name by your powershell in point 6.
    Nonetheless the “certutil -repairstore” command fixed the certificate with the missing privat key.

    Thank you,
    Marc

  2. Thank you so much, this helped me to “get back to work” after my servicedesk just told me to reset the whole client. Thank you for the wonderful script in step 7 and thank you for your time to investigate and share this. Your IT knowledge is magnificent. I want to add that maybe also checking for the Dmwappushservice” service and if it is running and on automatic could solve some folks problems. Also sometimes (in my case) for the double certificates it is not obvious which one is the zombie certificate (both were valid) so I exported the first one and deleted it and then tested (did not sync) then I imported it again and deleted the other one (now it worked).

    1. Hi! Thanks for the feedback! Yep.. the dmwap… is indeed important as I also mention here 😉
      https://call4cloud.nl/2022/11/the-device-with-a-clock-in-its-service/
      And the check is also build in to my intunesyncdebugscript
      https://call4cloud.nl/2022/10/intune-sync-debug-tool-the-last-royal-treasure/

  3. Your script runs clean on a PC I am working on but I am still getting the 0x80072f9A error and it is out of compliance in Intune. Any ideas?

    Specific error message is “MFM Session state is set to NoSession with last error “Exception from HRESULT: 0x80072F9A”

  4. I figured it out. I just deleted all the certs and then let your script fix it. Once it did that it brought the machine back into compliance.

Leave a Reply

Your email address will not be published. Required fields are marked *

4  +  1  =  

Proudly powered by WordPress | Theme: Wanderz Blog by Crimson Themes.