Autopilot & Pre-Provisioning’s Infinite Play…uh…Waiting list

Last Updated on May 10, 2024 by rudyooms

This blog will be no deep dive into a weird topic but more a warning when you decided to do something stupid but forgot about it….

I will divide this blog into multiple parts

  1. The Issue
  2. Troubleshooting part 1
  3. The ESP
  4. Troubleshooting part 2
  5. Oooopsssssieee

1. The Issue

Before showing what exactly broke, let’s start by looking at the issue itself. When I was writing my latest blog, which mentions the fake Autopilot@ and fooUser when using Autopilot for Pre-provisioned deployments, I stumbled upon some weird “Identifying” delay and decided to write a unique blog about it.

After the device preparation ESP phase, the device started the device setup part and needed to start the “Apps Identifying” part.

Please Note: This is a pure AADJ environment, so no evil or weird HAADJ issues here 🙂

Graphical user interface, application

Description automatically generated

Please Note: This “Identifying” issue could also occur when you configured the Enrollment Status page and configured the “Block Device use until all apps and profiles are installed”

This time it wasn’t failing because of a faulty configured ESP

As shown below the device was taking its time to identify all of those 7 apps that needed to be installed. After waiting exactly 30 minutes the “setup” key was created with all of the Win32 apps in it that needed to be tracked.

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\Device\Setup\

Text

Description automatically generated

At the same time, that “setup” key was created the device started installing the ESP-required apps. Installing those 7 small apps (Office 365 Apps weren’t included) “only” took about 36 minutes and I was shown a nice green Autopilot Sealing screen!

Graphical user interface

Description automatically generated

36 minutes… for a couple of apps… that ain’t right! Let’s dive into it a bit more

2. Troubleshooting part 1

I decided to start by running an ETL trace and while running the trace I also watched Fiddler go. When taking a closer look at the Fiddler trace, I noticed that almost every minute there was an outbound connection to r.maange.microsoft.com.

Every couple of minutes for about 30 minutes long it sends the same information over and over again and nothing more…

Graphical user interface, text, application

Description automatically generated

So I needed to take a look at the ETL trace but before I could do so I needed to add some additional providers to my WPRP file

As shown above, I made sure I added the configurmanager2 event provider with GUID “0ba3fb88-9af5-4d80-b3b3-a94ac136b6c5” to my WPRP file. Feel free to take a look at it.

https://call4cloud.nl/wp-content/uploads/2022/09/wprp.zip

After stopping the trace I noticed that the ETL file was a little bit larger than I expected but hey, I am not complaining. The bigger it is the more information it has right?

When opening this etl file in the WPA tool I immediately noticed a lot of “counts” in the Microsoft.Windows.DeviceManagement.Configmanager2 provider section

Graphical user interface, application, table, Excel

Description automatically generated

At first, I had the “stupid” idea it was also trying to list all of the Microsoft Store apps that were available before starting with the App identifying part but I guess that wasn’t the case! Let me show you why.

Luckily I am keeping an archive of all the traces I performed in the past… Because you can’t have enough of them! Just a couple of days ago I almost did the same thing on the same device, but that time I did NOT use the Autopilot pre-provisioning option but the normal user-based Autopilot enrollment

When opening this user-based Autopilot trace, I am noticing only a count of 16.000 rows instead of 95.000!

Image

So when comparing those 2, I was pretty sure it was doing “something” that was only occurring when running Autopilot for pre-provisioned deployments

3. The ESP Flow

I decided to do something funny. I started reading my own blog about the Enrollment Status (ESP) page, to hopefully spot something because it is “something” that is occurring in the ESP right?

Enrollment Status Page | Account Setup | Identifying | ESP (call4cloud.nl)

I guess I did… I even placed a note about it… Oh my….did I drink too much #membeer?

Graphical user interface, text, application

Description automatically generated

PowerShell scripts aren’t tracked during the ESP!!! So something not tracked could definitely give us some issues. But how, as I can’t come up with any reason why the PowerShell scripts could “only” give us issues when performing a pre-provisioning… or??

4. Troubleshooting part 2

I decided to run another pre-provisioning but this time I am going to add some PowerShell logging! To do so I enabled PowerShell transcription logging and configured the Output directory

Graphical user interface, text, application

Description automatically generated

With the PowerShell transcription logging in place, I started the pre-provisioning. Before I am going to show you the output of the transcription log file I also opened the task manager and the Intune Management Extension agentexecutor log file.

At the moment the ESP was busy “working on it….” I noticed that a PowerShell session in the system context was launched…. But NOT closed! It just stood there and did nothing!

Graphical user interface, text, application

Description automatically generated

Also when looking at the resource monitor, there was no disk activity at all! So I guess we got ourselves a lingering PowerShell session in the system context as it looks like. After closing the PowerShell sessions itself the device continued and started to install the Required Apps

After closing that PowerShell session, I decided to open the AgentExecutor log

The last line of this log was mentioning a specific PowerShell script It was trying to run. I opened the Endpoint portal and started searching for the ID

As shown below… ow my… that’s the Windows10_Bitlocker script I used when trying to come up with a bonkers solution

Device Health Attestation Flow | DHA | TPM | PCR | AIK (call4cloud.nl)

That’s odd because that script should only configure Bitlocker and would try to start encrypting the drive with the specified encryption during the device phase instead of waiting until some user logs in. Let’s continue to the PowerShell transcription log

Arrow, rectangle

Description automatically generated with medium confidence

As shown above, the device can’t be encrypted because the Active Directory Domain Services forest does not contain the required attributes and classes to host Bitlocker Drive Encryption or TPM information.

But…. It tries it again, again and again until eternity….

Funny Gifs : trippy GIF - VSGIF.com

Or when the PowerShell script times out after 30 minutes. Because 30 minutes is exactly the time a PowerShell script deployed to the device will time out. Guess what happens when the PowerShell script times out! Yes… the device will continue with the ESP and will start with the Win32app installations and will start tracking those apps

5. Oooooppps

First, some more explanation before I am going to show you the “ooooooopsssieee”. When you are enrolling a device with Autopilot Pre-Provisioning the device will be joined to Azure Ad with a fake autopilot@ account as I showed you in my latest blog

Autopilot Pre-Provisioning | Technical Flow V2 | Intune (call4cloud.nl)

After the device is enrolled into Intune, the Azure device certificate will be whacked and NO user will be logged in. Guess what doesn’t work when your device isn’t Azure Ad joined anymore and you aren’t logged in with a Microsoft Account?

Graphical user interface, application

Description automatically generated

When we have a nice NO longer Azure Ad joined device and we are not logged in with our Microsoft Account, it’s pretty obvious we can’t back up our Bitlocker Drive Encryption recovery information. You will notice a nice 0x801c0450 error in your Bitlocker-Api event log. Of course, when trying to encrypt the device manually with Bitlocker and trying to upload the key, you will also be prompted with a message that you can’t sign in with your Microsoft Account

Graphical user interface, text, application, Word

Description automatically generated

But who cares? Because you would expect that when launching that PowerShell script to encrypt the disk, it will just try to encrypt it and when it fails..uh the script will fail, right?

So I decided to fetch back the PowerShell script that was uploaded to Intune by using another PowerShell script

powershell-intune-samples/DeviceManagementScripts_Get.ps1 at master · microsoftgraph/powershell-intune-samples · GitHub

Guess what happened when opening that PowerShell script

A picture containing person, indoor

Description automatically generated

When taking a look at the PowerShell script I was playing around with an idea to make sure the Bitlocker recovery key is escrowed to Azure

If you combine the Waiting for / While part and the requirement to store recovery information in Azure Active Directory before enabling BitLocker without having an Azure Ad Joined device, guess what happens?

Damn… damn… stupid me! Okay, I didn’t break anything. Sometimes you try to fix something but end up breaking something else.

I decided to add a couple of lines to the Bitlocker PowerShell script (of course I could also convert it to a Win32app app or just remove those lines… ) I made sure the script first checks if the device is Azure Ad Joined before trying to encrypt the drive

Graphical user interface, text, application

Description automatically generated

After enrolling my device again with Autopilot Pre-Provisioning it only took 7 minutes instead of the 36 minutes we noticed at the start of this dumb blog

Qr code

Description automatically generated

Conclusion

Please be aware that when your device takes 30 minutes on identifying the Apps of the Device ESP stage, something is timing out! That “something” could very much be the PowerShell scripts which you are deploying to your devices!

Feel free to reach out to me if you are experiencing the same issue and it isn’t caused by your PowerShell Scripts

Should I Feel this Stupid? - The IT Hollow

7 thoughts on “Autopilot & Pre-Provisioning’s Infinite Play…uh…Waiting list

  1. But what if the pre-provisioning finishes in 10 minutes with 5 apps, but after sealing the device and assigning it to a user the device phase takes more than 50+ minutes for identifying apps? Nothing happens in the log after starting win32 app inventory collection via WMI and the next line after 17 minutes which says ‘set timer, start the timer for workload powershell and after another 30 minutes ‘co-mgmt features are not available, ex= system.Management.ManagementException, not fatal. Systems are Windows 10 21H2 and pure AAD joined, nothing hybrid?

  2. Hi Rudy,
    I have similar issue with the ESP – Device Setup – Application installation steps. In my case we are using the Windows Autopilot HAADJ with pre-provision. All the Apps that are part of ESP are Win32Apps including the O365 (Packaged as Win32Apps).
    Issue – Every now and then the application installation takes very long time. I have set the ESP time to 120 minutes of this issue. Sometimes it finishes and get the reseal within 40 minutes or 62 minutes or it timeouts after reaching the 120 minutes. Not sure which application is causing this, everytime i tried to do the autopilot diagnostics it will show different apps.
    Also, we are not using any powershell thing for bitlocker side. We have configured the bitlocker policy to enable via Disk encryption settings in Intune portal which will do the encrytion silently.
    Any suggestions what is going wrong with my ESP?

    Thank you,
    Sheetal
    Any suggestions on this?
    I

    1. Hi… Could be 1 from the 1000 issues… to get to the bottom, you will need to start looking at some logs… or sending me some… How many apps are configured as required in the esp list?

      1. Thanks for your message Rudy.
        It is 13 applications (including the Office365 Win32App) which are during the ESP stage. None of them has any dependencies. What logs you need?

  3. Hi Rudy,
    I have similar issues in our environment. we are using HAADJ with Win32Apps and it is taking very long time to get the reseal option. Sometimes it finishes in 32 minutes, or 60 minutes or 80 minutes or it times out. In our case we have set to 120 minutes for this reason.
    I had a check with my list of applications which are part of ESP and none of them has any powershell script.
    For Bitlocker, we use silent encryption which is configured via Intune Endpoint Disk Encryption.
    I am not that much familar with the ETL trace or fiddler.
    Any suggestions will be really appreicated.
    Thank you,
    Sheetal

  4. Also seeing a similar behaviour. We have a case open with Check Point FW, if you run without the CP inline ESP runs in approx 45mins. With CP inline it fails after the first app install, hangs around for over an hour and then we see a FIN sent from the workstation. The suspicion is the first app is a script and never gets to install the 2nd app. Some message is not getting back, it should be cut&dry firewall/packet capture analysis but we’re struggling. STu

  5. Hi Rudy, sorry for necroing this thread, but we’ve recently needed to change our BitLocker config from Intune, and we are using the script you reference in part 5.
    We have packaged it as a win32 app, and assigned it to all devices. How will the process become after making these changes? We’re in the process of testing an enrollment, but wanted to ask here as well.
    Right now we don’t have any BitLocker OS settings in Intune configured, but do everything in the script.
    Detection for this script is the existence of the task and some registry keys.
    Will this not encrypt the disk during the preprovisioning at all, and wait until the user ESP? Will it fail the application and then retry it again during User ESP?

Leave a Reply

Your email address will not be published. Required fields are marked *

  +  21  =  23