Last Updated on April 29, 2023 by rudyooms
This blog will be just me looking at the delivery of the EPMagent in my own troubleshooting style. While doing so I got focused on even something more important!! I will divide this blog into multiple parts
- Introduction
- Installation in 3 seconds
- First look with Procmon
- The WPR trace
- MMPC Part 1
- Fiddler
- Enrollments
- MMPC Part 2
Abusing the MS Docs!- Digging Deeper
- The Enrollment Flow
- KickStart it on your own
- What it did break…
Please note this blog is based on some fine-ass OSINT…. Not 100% sure if it’s totally accurate so changes can be made along the way. As I still need to look up more details, this is just version 1.0 of this blog
1. Introduction
I guess a brief introduction would be nice, right? In this blog, I had the intention to look at what would happen when you have activated Endpoint Privilege Management(EPM) and configured some policies in your tenant.
After we made sure to press “activate” the required DeviceHealthMonitoring policies will be delivered to your device.
With the DeviceHealthMonitoring policies being delivered to the device, the EPM Agent AKA MEMEPMAGENT magically appears on your device. Before going into many details, let’s take a look at what happens.
2. Installation in 3 seconds
When looking at the EPM installation we will notice that just before the EPM agent gets installed, “something” will trigger the EPM installation. The enrollment starts with the “Discovery Phase”.
In this discovery request, we will notice the Enrollment URL it needs to get to start the “Enrollment”. With the enrollment URLs in our pocket, it will try to fetch the Policies.
Repeat those 2 names… Discovery and getting policies… sounds familiar right?
As shown above, it kinda reminds me of the MDE enrollment? So I guess after the policies we should get the Binary Security Token? As shown below, after the device received the policies it would indeed receive the Security Token. Which is in fact the ???????
With the enrollment done, the SideCar Agent/ IME EPMAgent installation will be executed by the same CSP that also triggers the IME installation when the device gets enrolled into Intune
./Device/Vendor/MSFT/EnterpriseDesktopAppManagement/MSI
We will spot the same CSP when looking at the DeviceManagement-Enterprise-Diagnostics-Provider Event log
Of course, when looking at the EnterpriseDesktopAppMagement registry key, we can spot the same location being mentioned in the LocUri
Besides this LocURI we will also notice the real download location of the MSI file
After the discovery and enrollment phase, the CSP triggered the download. With the download in our pocket, it’s time for Msiexec to start installing this MSI file from the system its Local Appdata folder. It’s all system!!!
If everything checks out we have our wonderful Microsoft EPM Agent Service installed
With the installation of the Microsoft EPM Agent service, I guess the Blog is done!! Let’s drink some beer!
3. First look with Procmon?
I guess our journey is far from over. Let’s resume our journey to the center of EPM, by looking at the Procmon trace I got. Please Note: Besides this Procmon trace, I also got a nice Fiddler trace and some good old WPR tracing. So you are up for something nice!
In the Procmon trace, I filtered out the garbage and within a few seconds, I noticed the Epmservice.exe en taskhostw.exe every few seconds. Thread Create, thread exit. That can’t be a coincidence. The Taskhostw.exe, could indicate a scheduled task, and again looking back at the MDE enrollment that does ring a bell!
I guess that doesn’t sound that stupid. So let’s look at the Enterprisemgt scheduled tasks folder and the corresponding enrollments registry key. As in both locations, we should spot something funny?
Huhhh 2 Intune enrollments? that’s odd. That doesn’t sound good? Even the device name and EntDMID doesn’t match with the Device ID in Intune.? I know a thing or two about troubleshooting Intune issues and having a lingering Intune enrollment is bad ! (or isn’t it?)
I was curious if this enrollment also was levering the OMADMCLIENT.exe to sync stuff. Let’s switch back to our Procmon trace to find out. We will notice that with enrollment in progress, it indeed also launches the omadmclient.exe with some additional parameters
When looking at the additional parameters we will notice that the serverid corresponds to the Microsoft Device Management Enrollment… If we take a look at that main enrollment key we could also spot some “weird new” URLs
Okay we got ourselves an aadresourceid mentioning checkin.dm.microsoft and a nice discoveryservice URL, mentioning enrollment.dm.microsoft.com (instead of enrollment.manage.microsoft.com )
I guess with the time we are spending in the registry let’s take a look at the corresponding OMADM accounts registry key. In it, we will have another registry key sslclientcertsearchcriterea
Sounds familiar right? This SSL Client Cert setting is used to find the correct Certificate to be used for communication. So with this registry key, there should also be a Certificate? As shown below, yes! Besides the Intune device certificate, we have an additional certificate.
The Intune MDM certificate is used to secure communication with Intune so I assume this new Microsoft Device Management Device Certificate is also used to secure communication with “something”???? Not Intune I guess? I decided to take a look at my adjusted WPR file and the ETL trace it created.
4. The WPR Trace
So I combined some stuff from different trace providers and just put them all in a proper time frame
So before the MDMAppInstaller installs the EPMAgent, it first starts the enrollment (StartEnrollmentEvent) into something called MMPC? And after it got enrolled into MMPC, it got the device management certificate we noticed earlier. So I guess this device management certificate is used to secure communication with MMPC?
The MMPCENROLL part made me curious so I started opening the famous DLLs and EXE files that were updated when installing the 2023-03 preview update on my Windows 10 device.
omadmclient.exe, omadmprc.exe, deviceenroller.exe, mdmregistration.dll, omadmpapi.dll
5. MMPC part 1
MMPC…. It could ring a bell or two because it sounds like Defender. It could be the Microsoft Malware Protection Center but as that name is already taken it has to be something else. I guess it must be important as it Is mentioned in almost every DLL or other changed Executable file I open?
In a stupid attempt, I tried to use google and chatgpt. Somehow all my searches ended up in a complete failure
But I decided to spend some more time looking at the WPR trace I got. Looking back at my WPR trace and just looking at each tiny step it mentions Microsoft Device Management together with “Microsoft Management Platform Complete”
Mmm…okay… that sounds like a possible name except for the “complete” part. Besides the mention of MMPC, I also stumbled upon something I need to take a closer look at in the future when digging into the EPMAgent itself
As shown above, the DeclaredConfiguration is truly something I am not familiar with right now, so I decided to continue and take a look at the enrollment with Fiddler.
6. Fiddler
When looking a the Fiddler trace I got, once the EPMagent was installed (with the use of the exported Microsoft Device Management Certificate) I noticed some more details
As shown above, a nice ticket/token to authenticate at the checkin.dm.microsoft.com service we noticed earlier in the Enrollment registry. Could this be the “MMPC” I stumbled upon?
Besides the checkinuri, we will again notice the “DeclaredConfigurations”. These configurations are also stored in the registry. Please Check my blog about the WIN DC / Declared Configurations to find out what they are and why they are way way better
Epmagent | Declared Configuration | dcsvc.dll | Service (call4cloud.nl)
Mmm… okay that’s still not much to go with. At that exact point in time, I wanted to throw in the towel but then I suddenly noticed the POST in Fiddler
WCOS Checking… as in Windows Core OS checkin?
Okay, so besides the MMPC (which I still don’t know what it is) we also got WCOS. Keep that part in mind for now and let’s dig deeper
7. Enrollments key
So with some more new names and acronyms added to our brains, I switched back to the registry while looking at the two enrollments. The Intune enrollment suddenly had one nice addition.
In the Enrollment registry key, a new folder: LinkedEnrollment was created. As shown below, this LinkedEnrollment also contained some additional keys.
The picture above is telling us that our Intune enrollment is “Linked” with the Microsoft Device Management enrollment. Besides this information, It is also showing us the enrollstatus and something called MMPCLocked. I guess our Intune Enrollment is locked into the Microsoft Management Platform?
When taking a closer look at the deviceenroller.exe … Brainfart!! …
Brainfart mode ON
Am I even allowed to talk about the deviceenroller.exe???? As the first rule about the device enroller is…. We don’t speak about the deviceenroller.exe…
Brainfart mode OFF
When digging into a weird-ass named executable file, something caught my attention. “SetLinkedEnrollment”
Luckily (sometimes you need some luck) when using Google this time, searching for LinkedEnrollment, pointed me to a nice MS Doc.
As shown above, this doc is mentioning the fact that it will trigger a silent MMP-C enrollment by using the AAD device Token from the AADJ device. Luckily this “keyword” also pops up in some DLLs.
When spending some more time in IDA and looking at where it leads me, every time I end up with the same mention of “Dual Enrollment”. Maybe this could be a wild goose chase……but looking at the name “Dual Enrollment” we indeed noticed 2 enrollments. Double / 2 / Dual?
8. MMPC Part 2
But I am not giving up on my wild goose chase so I spend some more time looking at stuff and then I tried something else. Googling MMPC or WCOS alone gives you all kinds of results but putting them together showed me something beautiful
A principal engineer from Microsoft….. When looking at the details of the profile and her experience, we will notice some keywords in it
MMPC, WCOS, and reporting services… And the title…. Microsoft Managed Platform Cloud. So when just looking at the words and the Fiddler trace this is what I will come up with.
As we have 2 enrollments on 1 device. 1 Communicates with Intune and 1 with MMPC. But it seems that there is also some communication between them. How else does Intune know how to push the CSP to install the EPMAgent and where did you configure the EPM settings? Intune!!!
I also decided to look at my own Windows 11 Deviceenroller.exe from 07-05-2022….
Guess what I get when searching for MMPC ….
So the Microsoft Managed Platform Cloud is not something “100% new” and shiny but it is already in production for some time? Luckily using google on the dm.microsoft.com showed me a possible reason!
With Microsoft Defender for Endpoint (MDE), you can now deploy security configurations from Microsoft Intune directly to your onboarded devices WITHOUT requiring a full Microsoft Intune device enrollment.
I guess the only conclusion we could have: Microsoft Defender is already using the new Managed Platform! As it doesn’t use Intune it needs to use “something else”. That something else is MMPC
9. Abusing the MS-Docs
While writing this blog and learning new stuff every day just by looking at all sorts of code, Microsoft also added some new documentation. The first one is also mentioning the *.dm.microsoft.com.That URL does sound familiar indeed!
When looking at the “important note” and looking back at my own Fiddler traces.Yep… true… SSL inspection is not supported…..
By the look of it, there is a check if the MMP-C Server SSL Cert chains up to a known pinned certificate
The second MS-Doc addition mentions a “Microsoft Device Monitoring service”
10. Digging Deeper
After a good night’s sleep (NOT)…..
I decided to reinstall my device/vm but this time again without the 2023-03 preview and enrolled the device with Autopilot. When the enrollment was done, I took a look at the EnterpriseMgmt Tasks and noticed these tasks
I downloaded the 2023-03 preview update and rebooted the VM, but this time in a VLAN so I could open FIddler and capture some traffic. I opened the EnterpriseMGT tasks and first I noticed there wasn’t any additional task created.
Before removing the VLAN, I made sure I got the Intune certificate exported and attached to Fiddler
. With everything ready to go I removed the VLAN and “synced” the device. Within a minute or 2, the device started the “Discovery phase“. It did so by reaching out to the discovery URL (Discovery.dm.microsoft.com).
After the device was able to reach out to the discovery URL, an additional Task Schedule was delivered: “Schedule created by dm client for dual enrollment...to MMPC”
This scheduled task triggers the deviceenroller.exe with the /c /EnrollMmpc parameters! This task will stay on the device until its job is done! When you have Fiddler in its way or SSL inspection… the enrollment will fail! As shown below, you will receive a certificate chain error!
After “only excluding” the discovery.md.microsoft.com from HTTPS decryption, we will notice the enrollment kicks in.
Within this enrollment, we will notice that the EnrollmentType mentions MMPCComplete. Complete…. that is one thing I heard before…
Anyway, just look at the trace mentioning the same…
Once the device is registered for DualEnrollment to MMPC by using the AAD Device credentials it will delete the task that started the MMPC enrollment. Sounds a bit like cleaning up your mess 🙂
I guess besides the EPMAgent enrollment, I just found myself a new hobby….. troubleshooting “Dual Enrollment” With the MMPC enrollment, the real EPM installation kicks in and would start the installation of the Client Side Components / EPMagent.
11. The Enrollment Flow
When putting all pieces together we got when looking at all the flows/traces and available documentation, we will get this “enrollment flow”
12. KickStart it on your own!
I was curious if I could kickstart the MMPC / Linkedenrollment on my own. Why? Because it is supposed to be fun!
Besides the amount of fun it could give, apparently, it could happen that somehow even when you meet all the requirements the profile would still give you an error
As shown above, it only shows the “Allow Device Health Monitoring” setting run into some error. Nothing more! Microsoft their advice, for now, is to change the “Send Elevation data for reporting” to NO
Even with this setting from above, set to No, the EPMagent just isn’t getting installed somehow. I decided to do it myself!
So I made sure I got myself a Windows 10 device with a build that doesn’t match the requirements!
As shown below this Windows 10 22h2 build.
After making sure the device was enrolled with Intune and the initial sync was done I opened a PowerShell session and elevated myself to “SYSTEM” with the use of the psexec tool.
Once elevated, I first installed the localmdm powershell module
Once I installed that beautiful tool from some stranger called Niehaus, I created the same CSP I got from another great tool! The syncml tool!
With the <syncbody> in good shape, I made sure I first opened a new PowerShell session with the -MTA parameters
Within the MTA PowerShell session, I copy/pasted the sync body and fired off the send-request
$test1 = @"
<SyncBody>
<Exec>
<CmdID>12</CmdID>
<Item>
<Target>
<LocURI>./Vendor/MSFT/DMClient/Provider/MS%20DM%20Server/LinkedEnrollment/Enroll</LocURI>
</Target>
</Item>
</Exec>
</SyncBody>
"@
send-localmdmrequest -SyncML $test1Within a few minutes, the device got enrolled into MMPC, got the Intune Enrollment linked to the Privilege Manager, and got the EPM agent installed
I was wondering if Intune had the same feeling. So I opened the Intune portal and looked up the device, only to notice every required setting was marked as “succeeded”
13. What it did break(fixed)
After enrolling multiple test devices with Endpoint Privilege and getting them enrolled into MMP-C, something caught my eye when opening the Company Portal!
Somehow after opening the Company portal, it crashed and showed me the error: “An error occurred while attempting to load the devices”
I decided to look at the Company Portal log file in the user its appdata\local\packages\microsoft.companyportal_8wekyb3d8bbwe folder
Inside this log file, I noticed the fact, that somehow the device used the MicrosoftManagementPlatformCloudMdm (MMPC) agent
Where it normally should use the agent=Mdm and not the MMPC one
When opening Fiddler it should connect to the service by using the Intune Certificate that I attached but as shown below, it was indeed trying to connect to the MMPC MDM instead. I guess with the additional/linked enrollment, the Company Portal doesn’t know what to use!
To be 100% sure I decided to trash the Enrollments so the device would only be Intune enrolled and not MMPC enrolled. As shown below, after trashing everything only the Microsoft Intune MDM device CA was alive on the device
After the device was properly synced with Intune I opened the Company portal. Guess what was working again?
Hopefully, Microsoft will fix this issue very soon as having a broken Company Portal on all your EPM-activated devices, isn’t fun!!!
Looks like it started working out of a sudden… Still wondering how they fixed it 🙂
Conclusion
The Microsoft Managed Platform Cloud!!! Didn’t see this one coming! Besides the MMPC, what if we combine this with the WCOS check-in? AKA Windows Core OS AKA Windows 10x? In my understanding that project got ditched some time ago… apparently not!!!
It just feels like a honeypot! MMPC, LinkedEnrollment, DualEnrollment, WCOS Checkin
Stick with me and I will try to tell you more about the EPM agent next week!
Is the dual enrollment task only for autopilot or does it happen for Hybrid joined devices?
Great article… what is the workaround if you want to test EPM and still want to be able to use the Company Portal?
Great article, just opened a support ticket this morning because I was no longer able to open Company Portal on my 3 devices… I was testing EPM this week and yes, it looks like EPM did break Company Portal on my devices. I wonder how long Microsoft takes to find this out…
Ohh, being on the ball I see! Interesting read, especially for the device health tricky bit! 🙂
Hhm, interesting, but what would be the right way to prepare the target devices beforehand via Intune in regard of that Device health issue deployment error? That KB indeed helped me on some older 21h2, but I have a couple of 22h2 cases where even though the version is higher than requested, the error is happening.