Today, I visited a customer that I had installed an evaluation version of Configuration Manager at. 120 days passed a couple of weeks ago and now it was time to convert the evaluation into a full version.
Simple process really, insert CD, run upgrade … watch LUN on iSCSI storage array hosting the dynamically expanding VHDs fill up and cause the VM to stop cold. Hyper-V saves the state of the VM and marks it as critical, so no big deal, we just expanded the LUN. Unfortunately, in the process, the VM got powered off.
So, after expanding the LUN, we restarted the upgrade. Everything looked good — I stress “looked”. I created a new package, pushed it out to a test machine, and went to lunch. When we came back, the package deployed successfully, but I didn’t have any status for the new advertisement. Something was definitely wrong.
The compmon.log had the following over and over:
[ms_panel title=”” title_color=”#f5f5f5″ border_color=”#00274c” title_background_color=”#00274c” border_radius=”2″ class=”” id=””]
Can not get the current execution state for component SMS_SOFTWARE_INVENTORY_PROCESSOR since it is not installed or completed installing.Can not get the current execution state for component SMS_INBOX_MONITOR since it is not installed or completed installing.Can not get the current execution state for component SMS_DISCOVERY_DATA_MANAGER since it is not installed or completed installing.Can not get the current execution state for component SMS_STATE_SYSTEM since it is not installed or completed installing.Can not get the current execution state for component SMS_LAN_SENDER since it is not installed or completed installing.Can not get the current execution state for component SMS_INVENTORY_DATA_LOADER since it is not installed or completed installing.Can not get the current execution state for component SMS_WSUS_SYNC_MANAGER since it is not installed or completed installing.Can not get the current execution state for component SMS_MP_FILE_DISPATCH_MANAGER since it is not installed or completed installing.Can not get the current execution state for component SMS_SOFTWARE_METERING_PROCESSOR since it is not installed or completed installing.Can not get the current execution state for component SMS_SCHEDULER since it is not installed or completed installing.Can not get the current execution state for component SMS_STATUS_MANAGER since it is not installed or completed installing.
[/ms_panel]
Not good. First, I tried a site reset, no change. Then I tried the upgrade again, same thing. Time for more log spelunking.
The sitecomp.log had the following message for each of the above components:
[ms_panel title=”” title_color=”#f5f5f5″ border_color=”#00274c” title_background_color=”#00274c” border_radius=”2″ class=”” id=””]Installed service SMS_SERVER_BOOTSTRAP_SPFM-CM-01.
Starting service SMS_SERVER_BOOTSTRAP_SPFM-CM-01 with command-line arguments "SPF C:\Program Files (x86)\Microsoft Configuration Manager /deinstall C:\Program Files (x86)\Microsoft Configuration Managerbini386perfsetup.exe SMS_WSUS_SYNC_MANAGER"...
Execution of "C:\Program Files (x86)\Microsoft Configuration Manager\bin\i386\perfsetup.exe /deinstall /siteserver:SPFM-CM-01" on server SPFM-CM-01 failed: Child process exited with non-zero code 1000.
Bootstrap operation failed.
Deinstalled service SMS_SERVER_BOOTSTRAP_SPFM-CM-01.
Bootstrap operations aborted.
Reinstallation failed and will be retried in the next polling cycle.[/ms_panel]
Web searches at this point turned up nothing in particular. I did find a KB with a hotfix that was a stab in the dark: 954214. No change after installation.
I had looked through the Windows Application event log a few times before but didn’t act on anything there. I tried running the command line given in the sitecomp.log file manually and noticed that it created errors in the App log, so now it was time to look at them. The errors came in pairs:
LoadPerf 3011: Unloading the performance counter strings for service SMS_LAN_SENDER (SMS_LAN_SENDER) failed. The first DWORD in the Data section contains the error code.
LoadPerf 3012: The performance strings in the Performance registry value is corrupted when process Performance extension counter provider. The BaseIndex value from the Performance registry is the first DWORD in the Data section, LastCounter value is the second DWORD in the Data section, and LastHelp value is the third DWORD in the Data section.
Aside from the confusing English of these messages, I might have my culprit, corrupt performance data. Why this would cause the issue, I have no idea. I also checked PerfSetup.log and noticed the following, similar, errors.
[ms_panel title=”” title_color=”#f5f5f5″ border_color=”#00274c” title_background_color=”#00274c” border_radius=”2″ class=”” id=””]Error: UnloadPerfCounterTextStrings failed with command line 'Application SMS_LAN_SENDER' to unload performance counters; error = 1010 (0x3f2).[/ms_panel]
After a couple of web searches on LoadPerf 3011, I found the following command that resets performance counter data:
lodctr /r: c:\windows\system32\perf\stringbackup.ini
Amazingly, this worked and now all is good and right with Configuration Manager (again). I have no idea why corrupt performance counter data should cause such issues, but it did.