Keeping up with the EmmDeviceID And Config Refresh Bug

This blog will supplement my previous one, in which I explored the new DMClient LinkedEnrollment DiscoveryEndpoint Node. I will guide you through the EMMDeviceID rabbit hole I fell into, which led me to the Config Refresh bug!

1. Introduction

In one of my previous blogs (while on vacation in Luxembourg … ), I was trying to find out why my device wasn’t getting enrolled with MMP-C (EPM/Endpoint Privilege Management).

I discovered that somehow, we suddenly now need to configure a CSP to define the DiscoveryEndpoint.

https://call4cloud.nl/2023/11/in-the-shadow-of-the-discoveryendpoint/

If we don’t configure that one, the EPM enrollment AKA dual enrollment AKA MMP-C enrollment will fail

After adding the CSP, the first problem was solved. Yep, I thought the same until my device showed me some different kinds of errors during the same MMP-C enrollment process!

2. The Enrollment Issue

I stumbled upon a second issue with the DiscoveryEndpoint issue being solved by configuring an additional CSP. This issue started small but while looking at it, it became bigger and bigger.

As mentioned in my previous blog (DiscoveryEndpoint), I started looking back at the event logs, which now showed me that the Dual Enrollment task was successfully created. After this nice message, it started showing me some pretty bad errors.

As shown above, It showed me that it failed to enroll MMP-C for dual enrollment with the bad request 400 error. Okay so now we have a Dual enrollment scheduled task that can be launched but somehow the enrollment request is failing.

So, I installed Fiddler, exported the Management Certificate, attached it to Fiddler, and let Fiddler intercept the traffic.(capturing HTTPS traffic)

As shown above, with the discoveryendpoint CSP set, the required dual enrollment scheduled task is created and started. With the discovery service configured, it reaches out to the enrollment service (enrollment.dm.microsoft.com). We will notice that It fails at the last step!

As shown above, The last step in this process is getting the provisioning payload. This payload has the required Device Management Certificate in it. This device certificate is required to secure the communication between the device and the (MMP-C) service.

Somehow that last important step is failing. So, I decided to compare the failing enrollment with a fiddler trace I had from a working enrollment.

As shown above, the only real difference between them was the missing EmmDeviceID in the request. This EmmDeviceID is the Entdmid (enterprise device management id)

Looking back at the event log I got earlier, maybe that could explain why this value (NULL) was mentioned. Missing AKA NULL?

This Entdmid is pretty important. Without it, you can run into some weird certificate renewal issues. That’s why I added a check for that ID in the debug tool I wrote.

Intune Sync Debug PowerShell Tool to fix Intune Sync Issues (call4cloud.nl)

To be sure the EntDMID was configured and still on the device, I opened the registry, browsed to the current Intune enrollment, and opened the DMClient/MS DM Server key

As shown above, the registry key was indeed configured as expected. That’s weird, right? To make sure I wasn’t missing things, I opened Procmon, configured my filters, and triggered the Dual enrollment scheduled task to start the enrollment.

After capturing the Procmon trace, I started searching for Entdmid, and guess what it showed me!

Whoops… The dmenrollengine was looking at the wrong key to get the entdmid to pass it over to the request which was failing.

Without that entdmid being retrieved it just sends “nothing” and with it, we got a bad request because that entdmid is a requirement in the CreateEnrollMessage function as it seems.

Instead of looking at the MS DM server registry key, it was now looking at the ConfigRefresh registry key. Oh wait, I did configure the configrefresh policy some time ago to play around with it.

I even wrote a blog with a cool name about this new future!

https://call4cloud.nl/2024/02/configrefresh

Still, it’s weird that it is failing on the configrefresh key, right? So I decided to trash that configrefresh registry key and just remove the assignment from the ConfigRefresh policy

The first time I removed the ConfigRefresh registry node it didn’t work because somehow that ConfigRefresh registry node was recreated. Somehow it was recreated while I was removing the ConfigRefresh policy in Intune. So, I tried it again. After removing the ConfigRefresh key again, I immediately executed the scheduled task for dual enrollment.

This time it was completed without any errors! Great!!

3. The Chicken and the Egg

As mentioned in the previous part, I removed the ConfigRefresh Registry keys and executed the scheduled task for dual enrollment. Somehow the ConfigRefresh key was recreated but this time without any values in it.

The funny thing is that even with this key being empty, the DualEnrollment was still failing

Guess what recreated the ConfigRefresh registry key? Yep ConfigRefresh. What came first? ConfigRefresh Or ConfigRefresh

I deleted the keys, but of course, the corresponding scheduled task to execute the refresh was still there and still doing its job. So please beware that removing the registry keys does not disable config refresh. Now that is clear, let’s look at what exactly went wrong.

4. What went wrong?

Even while the enrollment succeeded (the second time) I decided to play around with it a bit more to make sure I wasn’t going crazy. I reinstalled the VM again but this time with the latest regular Windows build.

This time I did NOT configure the Config Refresh policy but I did something funny and exciting!

The enrollment registry keys were created just after the MDM enrollment was completed. Inside that enrollment key, the corresponding dmclient\ms dm server was also created. As shown below, after the enrollment registry keys were created and just before the MMP-C enrollment kicked in, I manually created the Config123 folder in it.

While enrolling the device and running Procmon, I noticed that the process wasn’t looking for the configrefresh folder this time. It was now only trying to find the entdmid in the config123 folder.

So I decided to take a look at the dmenrollengine.dll and MDMregistration.dll and started sniffing around. After a single search for entdmid, I came across this function: GetEnrollmentEntDmID

When looking at the code, it tries to enumerate (regenumkeyexw) the underlying reg keys in the DMClient registry folder.

After it enumerates the subkeys, it will only fetch the FIRST subkey it finds (hkey, 0 )

I guess with the C being placed before the M in the alphabet, it just gets the first possible key and that’s the C? From the first subkey it got, it will try to get the EntDMID

Mmm, this sounds and smells pretty bad. Having issues with the EPM enrollment is one but….

5. What Else could it break?

With this EPM Enrollment issue, I was curious about what happens when our Intune Device Certificate needs to be renewed. Does it also need that EntDMID key? Looking at the MS Docs, I guess it does.

Let’s find out, shall we? I tried changing a couple of registry keys to ensure that the device would start contacting the service.

After changing the RenewalPeriod, removing the RenewTimeStamp, and running the schedule to renew the certificate, the device contacted the enrollment service.

While communicating with the service, the deviceenroller.exe also tried to find the entdmid in the WRONG registry key! that can’t be good!

So if “some” registry key is created inside the DMClient registry folder that starts with a letter before the M, we are screwed! Asking for the EntDMID myself with the localmdm PowerShell tool showed me the same error! 404 Not found!

Linked/Dual Enrollments will fail! And probably also the certificate renewals will fail! Everything that needs the entdmid will fail! This is bad…??.

6. The Flow

As always, the flow—this time, not in paint. 🙂 Please reply if you fancy this flow more than the paint one.

7. What I did

After discovering that deploying ConfigRefresh could lead to serious issues, I contacted some of my dear friends at Microsoft. The time difference between Redmund and the Netherlands is pretty bad, so I needed to wait until they got out of bed 🙂

After explaining what I noticed, they took it pretty seriously, and after a while, it was obvious that the bug was indeed nasty and needed to be fixed!

A week after getting a good explanation of what Microsoft would do to fix it, I noticed a slight change in the Settings Catalog. This is how it looked like before.

The moment you create a new Settings Catalog and select config refresh, you will notice (when I wrote this) that it was now telling you (Coming Soon)

Microsoft also ensured that the setting didn’t apply to all of our devices by configuring the minimumsupportversion to 99.9.9999.

But that’s not all.. when stumbling upon this nast ConfigRefresh/EntdMID issue in the last week of October, the possibility to Pause Config refresh with a remote command was just added to the RemoteTasks documentation

Guess what happened a week after the issue was reported? The ConfigRefresh Remote Pause task that was added to the Remote Tasks documentation was “reverted” aka Removed from the docs (long live Github).

Until Microsoft releases the fix, it seems they pulled back ConfigRefresh! (Which was a very smart thing to do!!!!) If you watched Microsoft Ignite, it looks like the fix for ConfigRefresh will be released sometime in Q1 of 2024, and with it, we will be able to deploy ConfigRefresh again! Whoohoooo!

8. How Microsoft Fixed it

With the latest Windows insider canary 26040 build and the latest March Windows 11 Update KB5035853 KB5035854, I noticed a slight change in the enterprisecsps.dll. Instead of creating the ConfigRefresh registry key inside the DMClient, Microsoft moved it to the root of the enrollment instead.

With this MoveConfigRefreshPath, they made sure the ConfigRefresh key wasn’t interfering with other processes that needed to fetch the proper Entdmid in the first subkey of the DM Client registry key.

So by the looks of it, Microsoft changed some code in the enterprisecsp.dll instead of ensuring the proper MS DM Server registry key was enumerated to find the Entdmid in the dmenrollengine.dll. As shown below, it changed a bit but SFAIK it still looks for the first registry key inline

Conclusion

With the Entdmid issue fixed, we can now start deploying and using Config Refresh for many of your Intune policies! But I love the schedule to refresh any setting on the device (declared Configuration) way more!