Willy’s White Glove Autopilot Wonderland

Willy’s White Glove Autopilot Wonderland

This blog will be about my experiences with Autopilot for pre-provisioned deployments (White Glove) and I will try to talk you through the whole process.

You might ask, why did he write it? Some time ago I was troubleshooting the Autopilot white-glove deployments because there were a lot of issues with the TPM attestation part (0x80180014 and 0x800705b4), while troubleshooting I noticed some weird stuff (in my opinion) as I can’t find any official Microsoft documentation about it.

I will divide this blog into multiple parts

  1. A Brief overview
  2. Uploading the Hash
  3. Starting the Autopilot pre-provisioning
  4. The ESP preparation Phase
  5. The User Login
  6. Tools Used

1. A Brief Overview

Before I am going to try to show you the whole flow. I will need to show you a short summary of what happens in the background when you are enrolling your device into White Glove after you uploaded the Device Hash into Intune.

Looking at the picture above, you would think that not too much is happening right? A little bit of an Azure Ad Discovery, a little bit of an Azure Ad Join, enrolling the device into MDM and Unjoining the Device…. Wait?? What? Unjoining the device?

In the next few parts, I will try to explain it a little bit more. Also, it can be useful to read this blog about the Device Health Attestation process first…As it contains some words and explanation you could need later on

2. Uploading the Hash

Of course, we need to make sure we have uploaded the device hash before proceeding.. so go ahead and upload it with the get-windowsautopilotinfo.ps1 Powershell module

I am going you to methods to check if the hash has been uploaded and your devices are ready to go

  1. Endpoint Manager Portal
  2. PowerShell or Graph Explorer

2.1 EndPoint Manager Portal

First, let’s open the Endpoint Manager and browse to the Autopilot section. Search for the serialnumber and you will notice the hash has been imported

Did you read the DHA blog I mentioned earlier? To summon it up, (Technical details will follow) when you uploaded the device hash to Intune you sort of added the EKPub to an allowed list of EKPubs that are permitted to perform Attestation.

Afbeelding met tekst  Automatisch gegenereerde beschrijving

Please note: At the same time you uploaded the “hash”, the Azure AD Registration Service (DRS) will also pre-create the Azure Ad object.

While pre-creating the object it will also get the famous ZTDID (Zero Touch Device ID) value attached to it.

But beware when the device is pre-created, the device object will still be configured in a disabled state

2.2 PowerShell or Graph Explorer

Just like always, you can also check out your imported Autopilot devices with Graph. To do so you will only need to open the Graph Explorer (or PowerShell )

When you have opened the Graph explorer, please make sure you have the proper permissions and consent to them, to fetch those devices.

Now put in this URL and watch the magic happen! You will notice that this GRAPH URI, shows you the history of the imported Autopilot Devices. I guess the URI: importedWindowsAutopilotDeviceIdentities just says enough…

(Please note: I don’t know yet for you long it keeps its history)

https://graph.microsoft.com/beta/deviceManagement/importedWindowsAutopilotDeviceIdentities

As shown above, it is perfect when you need to troubleshoot the DeviceImportStatus or the DeviceErrorCode!

3. Starting Autopilot Pre-Provisioning

Now it’s time to start the Autopilot provisioning part. You could do so by pressing the window key five times. By pressing it 5 times, the “CloudExperiencehost” will leave the normal OOBE screen and you could start with the provisioning part.

When clicking on “next” it will start downloading the Profile (AcquireProfile) if it’s available or not already done and will check if there are any Zero-Day Patches that needed to be deployed. I guess it’s very important…your device is always up to date and very very secure.

After it checked if there were any ZDP available it will show you the famous Autopilot landing page to start provisioning. (PerformDeviceEnrollment)

Please note: This whole provision progress will be done within this user context: Defaultuser0

4. The ESP Preparation Phase

Let’s take a look at the ESP Preparation Phase. If you want to know more about the whole flow, I already did a blog in which I am also mentioning the Enrollment status Pages.

But I need to go a little bit deeper with this, while dealing with White glove/Self-deploying mode we need to take a better look at the Device Preparation phase so I will also divide this part up into multiple subparts

4.1 Device Preparation (Securing hardware)

4.2 Device Preparation (Joining your organization’s network)

4.3 Device Preparation (Registering for MDM)

4.1. Device Preparation (Securing Hardware)

After the device preparation phase, the device needs to use the Endorsement Key (EKPub/EkCert) from the TPM for attestation to authenticate to Azure AD. But before this attestation process can be performed, the TPM needs to be verified and trusted.

I can’t say it enough! This is one of the most important parts and a lot of white-glove errors are due to this part. Always make sure your device has a TPM 2.0, supports Attestation and make sure your device is totally up to date with the latest Windows 10 Build. Let’s go further.

So when you don’t have a device with a discrete TPM but a Firmware TPM you are up for a treat!. Because these firmware-based TPM devices don’t include all needed certificates at boot time.

We have only 3 TPM suppliers who manufacture firmware TPM devices: Intel, AMD, or Qualcomm

These devices must be able to retrieve those certificates from the manufacturer (some from an internal ODCA and some from the vendor itself) when you are performing a white glove.

Part 1: Retrieving the EKCert

Again it’s all about trust, so the TPM needs to get proof that it can be trusted. We all know how you can build up trust! By using certificates. So let’s get the one we need….(as a firmware-based TPM doesn’t have the needed certificates preinstalled like the discrete TPM)

To get this certificate, the AIK request process will kick in (TPMTaskUpdate) to start the SCEP certificate enrollment.

After the Certificate Authority (Intel: https://ekop.intel.com) received the certificate request it will set up a secure channel with the client and will ask the client for more details to prove its identity.

But before proving its identity the client needs also to validate the CA public key as it is a two-way trust. The client will first check if the Public CA is also trustworthy. If the CA is trustworthy and valid, the client will answer the call by sending the endorsement credentials to the CA to prove its identity

*Endorsement credentials contain the Public EK (EKpub) and information about the security properties of the TPM

With this information, the Public CA will check if the TPM is not at risk (Revoked/Compromised). If everything is good to go it will create and sign it with a certificate (*EKCert) to prove that the EKpub is associated with a genuine hardware TPM.

*The EKcert contains the EK public key and is signed by the manufacturer.

After the client received the certificate encrypted with the EKPub part (EKCert), it can use this certificate to start the attestation. But still, an important part is missing.

Part 2: Establishing a connection to the Microsoft AIK Service

So now the device needs to establish a secure connection with the Microsoft attestation service (*.microsoftaik.azure.net) to get back the Attestation Identity Key certificate (AIK). It will do so by sending the AIK and the EKCert to the Microsoft CA. Now the Microsoft CA is 100% sure it’s communicating with a real living TPM it will send back the so much needed AIK certificate!

Afbeelding met tafel  Automatisch gegenereerde beschrijving

This AIK certificate will be used to provide the necessary identity trust / platform-based authentication.

Also good to know is, that in this AIK certificate, you will find a special OID. Its job is to certify that the endorsement certificate was used during the attestation.

This process (Getting back the AIK certificate) will be retried 10 times if it could not reach the CA you could end up with a *GetCACaps 404 not found or a 400 bad request in the CertReq_enrollaik_Output.txt.

*GetCACaps: This message requests capabilities from CA. The response is a list of text capabilities

Afbeelding met tekst  Automatisch gegenereerde beschrijving

This issue can be due to the fact you are not running the latest Windows 10 build or for example a DNS problem (it’s always DNS?) Luckily with 21h2 the TPM attestation process when you have tiger lake devices will be solved….? But I guess that’s something for another blog! (as testing with 21h2 did not resolve it)

And with this step, the first phase (Securing Hardware) is done. Even when it’s normally done within a few seconds, a lot of stuff happens in that few seconds.

4.2.Device Preparation (Joining your organization’s network)

After the Certificate has been received, the TPM identity is confirmed and the trust relationship is built, a Win32 API call will be triggered to start the discovery operation (AADDiscovery). This Win32 API call is implemented in the dsreg.dll.

Please Note: This DLL file is very important as it is also responsible for the DeviceJoin / Unjoin. So every time such an operation will occur, you will notice the dsreg.dll will be called upon.

Afbeelding met tekst  Automatisch gegenereerde beschrijving

It will now start to get all Azure DRS and device management endpoints it needs to have before proceeding.

Now we have everything we need the TPM will use the EKPub during attestation to prove to Azure Ad that it’s not an imposter and that it’s the same device that matches the hash you imported earlier. Why I am mentioning the hash? Because the hash also contains the TPM EkPub, so it can compare those 2!

When the EKPub’s from the device and Azure ad have a match the device will need to prove its identity. This is done with the EKPriv and setting up a secure challenge (nonce) used to prove its identity

When the device is properly authenticated with Azure Ad, it will send back the Azure Ad device id token. This device token will be used to join Azure Ad later on (JoinDevice) and it will also be used to enrol into Intune. So with this, the device registration begins. An event 102 will be logged with the message: JoinRequest: 11 (device_auto_ddid)

Afbeelding met tafel  Automatisch gegenereerde beschrijving

At this moment the RSA key pair (device key / DKPub/DKPriv) will be created and will start requesting the device certificate with the use of the DKPub and will sign it with the DKPriv.

(I guess that’s why I mentioned the DHA blog earlier… it will show you some insights about attestation)

But while creating the RSA key pair an additional key pair (storage transport key /TKPub/TKPriv) is also created. This key will be used to make sure you could authenticate to Azure Ad later on. Of course, all of these keys are stored in a nice secured place, the TPM.

After all this information has been gathered, a device registration request containing the storage/transport key /device token/attestation data will be sent to Azure DRS.

After Azure has validated the token, it will generate and send back the device certificate

Please note: At the same time the Azure Ad device certificate arrives at the device, Azure DRS will also enable and update the pre-created device object.

If you are fast, you are going to notice something strange and not documented at all?.

Afbeelding met tekst  Automatisch gegenereerde beschrijving

For just a few seconds you will notice the device cert is still in the Computer personal store and the device will become Azure Ad Joined. Why? Because it’s a necessity for enrolling the device into Intune!

At this moment it will try to get the Access token (ESTSTicket) to start the Intune Enrollment

4.3.Device Preparation (Registering for MDM)

When the device is successfully Azure Ad Joined, the MDM enrollment will start. To successfully enrol into Intune, it will need to have a token to authenticate against the Intune enrollment service to start the MDM enrollment (MDMEnroll)

Afbeelding met tekst  Automatisch gegenereerde beschrijving

Luckily when the device was successfully enrolled into Azure Ad, the device stored the device token in its cache. This token can also be used to authenticate against the Microsoft Intune service to retrieve the Microsoft Intune Device Certificate.

But just a few seconds after the Intune MDM device cert arrives the RegistrationCertHelper process will start and the Device certificate will be whacked.

It happens just a few milliseconds after the MDMEnroll took place.

And you all know what happens when the device cert is removed! The device will be unjoined instantly!.

When again looking at this flow, you will notice the AutoPilotManager will trigger the device to unjoin Azure Ad after the certificate has been removed. You should expect this only to happen with HAADJ like is mentioned in this MS Article

But I guess it’s also done with a normal White Glove flow. When the certificate has been removed and your device is unjoined a nice event will be logged Device_unjoin (UnjoinDevice).

You could check it out for yourself. Open PowerShell and you will notice the device will no longer be Azure Ad Joined

Afbeelding met tekst  Automatisch gegenereerde beschrijving

Of course, there is still some cleanup to do, so some “stuff / Keys” from the defaultuser Microsoft.AAD.BrokerPlugin folder will be removed

But even while the device isn’t Azure ad joined anymore the MDM enrollment will still go fort.(luckily )

While preparing your device for mobile management, you will also notice the device is reaching out to the Office 365 CDN to start downloading office 365 even when it’s not in the device setup phase! (if the Office 365 apps are configured in your tenant as a required app)

Afbeelding met tekst  Automatisch gegenereerde beschrijving

In this blog, I am only focusing on the first preparation part as the device setup phase speaks for itself, but I still needed to mention the Office CDN part as a lot of people are not aware that this is happening before the device setup phase.

5. User Login

After the device is sealed with a nice green screen, it can be handed over to our employees. When the device boots up, it will check if the device is already provisioned. So it will skip this step if it has been provisioned successfully earlier.

So we end up with the nice login screen after again checking for ZDP, which prompt us to log in. Please note, we are still running the whole process with the DefaultUser0 user context. Normally when logging in, it will need to start with the last ESP phase: Account Setup.

Looking at the Device Preparation flow, you might have noticed the device isn’t Azure Ad joined anymore so how could we log and still get our PRT?

In a normal situation, turning on the device will boot into the Microsoft Account Sign-in page (WWAHOST.exe). In this Sign in with Microsoft screen, you will be prompted to enter your credentials. When entering your credentials at the first login screen, an authentication buffer will be created.

I guess I still need to learn all the details of this process, so I guess this deserves a blog of its own later on….

Logonui will send this authentication/credential “buffer” to the LSA. LSA it’s job is to send it to the *Cloud Authentication Provider.

*Cloud Authentication Provider (CloudAP): CloudAP is the modern authentication provider for Windows sign-in, that verifies users logging into a Windows 10 device. CloudAP provides a plugin framework that identity providers can build on to enable authentication to Windows using that identity provider’s credentials.

So, after the user has entered his/her credentials, the logged-on user “defaultuser” will be logged off and the CloudAP provider will authenticate the user and the device with Azure Ad and it will again start the whole device registration again! But this time with some nice user credentials.

Afbeelding met tekst  Automatisch gegenereerde beschrijving

So again the DRS endpoint discovery will start and Azure Ad will validate the user credentials, verify if the device is valid in the tenant and will send back the encrypted PRT together with the *Session key using the Transport key which I showed you earlier!

*The session key is your Proof of possession for additional requests with the PRT

The CloudAp plugin will ask the TPM to unencrypt this session key and will store the session key, together with the PRT in its cache. Now take a look at your certificates, the lost Azure AD device certificate will be brought back to life again and your device is again Azure Ad Joined!

At this same moment, it will also push a device configuration change to azure to configure the owner of the device. This will be your user by which you enrolled the device and not the defaultuser0 of course…

So resume: At the first login screen, an authentication buffer will be created to join Azure Ad. This buffer doesn’t like reboots as it is only valid for the deployment itself! So please, pretty please when your White glove screen presents you with a nice green screen, just seal it and don’t touch it anymore! Just send it to your employees

If you are interested in what could happen when you don’t listen to my advice, please visit this other blog about Autopilot and the lost Azure Ad Join

6.Tools Used

I want to finish off this blog by mentioning the tools I used to get a good understanding of what was happening during white glove

1.WireShark to try to export the client certificates needed for the HTTPS decryption

2.MDMdiagnostics tools to export the needed event logs (User Device Registration, Moderndeployment-Diagnosticprovider/autopilot, AAD, Shell Core)

3.WPR tool to enable event tracing during autopilot and imported it in the Windows Performance Analyzer (I will create an additional blog for this one! As it needs some more explanation!)

And here it is… The blog about how to use the WPA to start some troubleshooting

https://call4cloud.nl/2021/11/theres-someone-inside-your-etl/

4.Sysmon with everything on exclude… so it will include everything

5.Fiddler:

Conclusions:

It’s always smart to get a good understanding of what happens underneath. I am still not sure why the device join and immediately unjoin NEEDS to happen when not using HAADJ. I guess it can be due to 3 reasons.

  1. The device owner needs to be set/configured (as it can’t be changed with graph… only the primary user can be changed). And the only way to do this, is with a new Azure Ad join?
  2. The added it for the ADDS join, but while adding it.. it was also added for the normal white glove flow?
  3. Security related and I am not authorized to hear why?

Also looking back at an earlier blog… I guess I now know the reason why it just broke… after the green screen we sealed it… but powered it back on to make sure the device was totally up to date by running the windows updates from the first OOBE prompt. Something is 100% sure… the authentication “buffer” doesn’t like reboots!

The authentication buffer…. I am still intrigued by this one…

As there must be some kind of buffer to make sure when the device is powered on, you will be redirected to the company login page.

If you want to read more about TPM attestation, please check out my attestation series

Attestation and Compliance Series – Call4Cloud

3 thoughts on “Willy’s White Glove Autopilot Wonderland

  1. This is handy information. I am working on this for my organization and encountering MDM enrollment failures due to the TPM chip and the URLS for ekop.intel.com/ekcertservice and *.microsoftaik.azure.net are blocked. Where in Azure can these URLS be added/whitelisted to get this working?

Leave a Reply

Your email address will not be published. Required fields are marked *

25  +    =  31