Updates
-
Updated May 8 to add information on a second issue.
-
Updated May 9 to add information on possible additional issues as well as instructions for using the recovery console.
-
Updated May 10 with some clarifications, a possible video driver problem causing other STOP errors, and an additional work-around for the ASUS motherboard.
-
Updated May 11 with a pointer to a Microsoft article on removing SP3, and added some information on a possible version for the faulting ATI Catalyst driver.
-
Updated May 12: Added information on free support, and a note on Media Center Digital Rights Management problems.
-
Updated May 13: Added some information on how to determine which control set to modify for the intelppm workaround. Also added a pointer to an HP support article on the problem and a request to verify a claim made in that article
-
Updated May 14: Received confirmation about how HP configures its computers. Added an explanation to how the problem occurs.
-
Updated Again on May 14: Not sure why I didn't think of this until now, but I
wrote a small tool that will detect the IntelPPM problem and mitigate it before installing the service pack.
-
Updated May 20: Fixed the description of the intelppm.sys problem to more accurately represent how the problem occurs.
-
Updated May 22: Added a note on how to properly download the tool using Safari.
-
Updated May 24: Added information on conflicts with anti-malware software, including Symantec's suites. The short version is: you MUST disable any security software before installing SP3.
-
Updated June 4: Added information on a conflict with certain wireless cards.
Before you read on, read this!
There are several issues that can cause a Windows XP computer to not reboot properly after installing Service Pack 3. Most of them affect relatively specific configurations, and most appear to have relatively simple work-arounds. Please: do not do anything rash. I have seen a lot of reports of people who reformat and reinstall when they run into this problem, losing all their data in the process. There is often no need to do anything that drastic. First read this post, and see if anything here helps you. If not, call Microsoft's technical support line and see if they can't help you.
If you have not yet installed SP3, make sure you disable, or better yet, remove, any anti-malware suite before doing so. If you do not, it is possible that you will get various kinds of corruption during the installation.
Free SP3 Support from Microsoft
EmilySc, a Microsoft employee, posted in the newsgroups yesterday that there is now free installation and troubleshooting support for SP3. This may be a real help to those who need interactive help solving the problem.
You can find all the support options on the Microsoft Support Website. In North America, free telephone support is available by calling (866) 234-6020.
The Problem
Last night WSUS deployed XP Service Pack 3 (SP3) to the sole remaining computer running XP that I have. This morning, I came down and was greeted with incessant reboots. The computer booted, apologized for not being able to boot properly, asked if I wanted to boot into safe mode, defaulted to normal boot, rebooted, and so on and so on. At this point, I want to clarify that the endless rebooting is not at all related to SP3 per se. The problem is that with some configurations, SP3 causes the computer to crash during boot, and Windows XP, by default, is set up to automatically reboot when it crashes. That is why you end up in the endless rebooting scenario.
There are many possible reasons why a computer may crash at boot time. SP3 seems to introduce two that are related to AMD-based computers, and, possibly, one or two more that appear to affect Intel-based computers. Which one it is impacts which work-around you use. At this point, the information is still trickling in. If you have a crash on boot problem that does not match what I describe below, and it happened as soon as you installed SP3, I'm sure others would like to know as well, including as much detail as you can give us.
First problem, affecting AMD-based computers with OEM images, primarily HP Desktops
NEW: Use this tool to mitigate the problem
If you have an AMD-based computer, and all you want to do is prevent the problem before installing Service Pack 3, then try the new tool I just wrote. It will first check whether you have an AMD-based computer. If you do it will check whether the IntelPPM driver is set to load. If it is it will offer you an option to disable it. The tool works by simply double-clicking it. If you need to check many computers on a network you can do that by running it from the command line, using this command:
removeIntelPPMonAMD.vbs <computer 1> <computer 2> <computer 3>...
It will take an arbitrary number of computers. The only caveat is that the tool will prompt you several times for each computer. If you really need a silent version, I can probably be persuaded to write one for you.
Note that if you are downloading the tool on Safari there is a bug in how Safari handles these types of downloads. If you just click on the link Safari will save the tool with a .txt extension instead and open it. You can remove that extension and then double-click the tool to run it. If you right-click the link and select "Download link as..." Safari will put the name on the containing page on the tool, not the name of the tool itself. You would need to rename it to something with a .vbs extension first to use it. Neither Firefox nor Internet Explorer makes it this difficult to download that tool, although Firefox does not properly handle right-clicking and selecting "Save link as..."
Disclaimer: the tool is provided "as is" with no warranty express or implied. It is designed to make changes to your system and those changes always carries a risk. Even though I have tested it as much as I can, I cannot guarantee that it will work for you. By running the tool you agree to hold me harmless for any damage it may cause to your computer.
Problem Details
In my case, the computer would boot into safe mode fine, so I did that. Not knowing what it was, I ran a disk check, which turned out to be a real mistake. Once I configured the computer to run a disk check at startup it would not even boot into safe mode.
Fortunately, I know Bill Castner, another Microsoft MVP, and he pointed me to a solution. It turns out that this computer is running an OEM OS image from HP. If you have an HP computer with a part number that ends with a 'z' you have an AMD-based computer. Other manufacturers have also shipped AMD-based computers, but it is unclear whether they have built their images the same way HP did.
The problem is that HP, and possibly other OEMs, deploy the same image to Intel-based desktops that they do to AMD-based desktops. It also appears that this is unique to their desktop image, and any HP AMD-based laptops are unaffected by the problem. Because the image for both Intel and AMD is the same all have the intelppm.sys driver installed and running. That driver provides power management on Intel-based computers. On an AMD-based computer, amdk8.sys provides the same functionality. Microsoft points out in a Knowledge Base article that installing both drivers on the same computer is an unsupported configuration, putting the blame on the OEM that deploys the image. The article in question was written when the same problem occurred after installing Service Pack 2 for Windows XP.
Ordinarily, having intelppm.sys listed in the registry on an AMD-based computer appears to cause no problems, so long as the binary does not actually run. On HPs images, the driver is not installed, even though the driver is listed in the registry and supposed to load. However, on the first reboot after a service pack installation, it causes a big problem. The computer either fails to boot, as in my case, or crashes with a STOP error code of 0x0000007e. If you see that error code you almost certainly have this problem. The computer will boot into safe mode because the drivers are disabled there. Please note here that simply having the intelppm.sys file on your computer is not the problem so searching for it in the Windows directory is not relevant. Nor is only having a directive in the registry to load it a problem. It must be running to cause a problem, which means the file has to both exist on the disk, and the registry has to be configured to load it. Therein lies the problem. HPs images have the registry key set but no driver on disk. When the service pack is installed the pre-existing directive in the registry is read, the installer lays down the driver on the disk, and on the next reboot it launches, causing the crash.
You may not see the error code because the computer reboots too fast. To force the computer to stop when it crashes, you need to set an option during startup. To do so, hit the F8 key during restart right when you see the black Windows XP screen come up. Then select the "Disable automatic restart on system failure" option, as shown below:
To fix the problem, boot into safe mode, or boot to a WinPE disk, or into the recovery console, and disable the intelppm.sys driver.
WARNING: Do NOT under any circumstance disable the intelppm driver on an Intel-based computer. It will make your computer not boot! If your computer will not boot because you disabled the intelppm driver on an Intel-based computer, follow the directions in the Recovery Console section below.
If you have an AMD-based computer, however, you do not need the intelppm driver and can disable it. Boot into Safe Mode by hitting the F8 key as above, but select Safe Mode instead. You will need your Administrator account to log on in safe mode. To disable the driver, take the following steps:
If you booted into the recovery console, from a command prompt, run "disable intelppm"
If you booted into safe mode you can run "sc config intelppm start= disabled"
If you booted into WinPE, you have to manually edit the registry. Do this:
-
Run regedit
-
Click on HKEY_LOCAL_MACHINE
-
From the File menu, select "Load hive"
-
Navigate to %systemdriver%\Windows\System32\Config on the dead system and select the file name System
-
Name it something you can remember, such as "horked"
-
Navigate to horked\<the current control set>\Services\IntelPPM. See below for how to determine which one is the current control set.
-
Double click the Start value and set it to 4
-
If you did what I did and completely destroyed things by running a disk check, navigate to <the current control set>\Control\SessionManager. Open the BootExecute value and clear out the autochk entries
-
Reboot
Step 6 asks you to navigate to <the current control set>. Under HKEY_LOCAL_MACHINE\SYSTEM there are typically at least two numbered control sets, and sometimes there are up to four. They are called ControlSet001, ControlSet002, and so on. Control sets hold all the configuration data for the computer, including all drivers that load. One of them is designated the current one, and the others are backups of previous configurations that worked. The control set that is currently used as the current one is the one listed in the "Current" value under HKEY_LOCAL_MACHINE\SYSTEM\Select. That is the control set that you need to modify in step 6. If you modify one of the other control sets it will not solve the problem. You need to modify the current one. If you manage to boot the computer, CurrentControlSet will be a pointer to the current one and you can modify that one. If you boot from the recovery disk you have to figure out which one is the current to modify the proper one. It will not always be ControlSet001.
If this was your problem, the computer should now reboot just fine.
HPs Response
On May 13, 2008 HP posted a support article on this problem. In that article they claim that the Service Pack copies the intelppm.sys driver to the computer even though it was not there before the Service Pack was deployed.
HP is partially correct. On their desktop images the intelppm.sys file does not exist in the %systemroot%\system32\drivers directory prior to installing the service pack. However, on its laptop images the file does exist there. By contrast, on the HP desktop images the intelppm registry key does exist under HKML\SYSTEM\CurrentControlSet, and it directs the driver to start. On the laptop images, the registry key does not exist. This is why HPs desktop images exhibit the problem and the laptop images are fine. It is not the presence of the driver on disk that is the problem. It is the instruction to load it that HP put into the registry that causes the problem.
That would also explain why the SP3 installer lays down the driver file on disk even though it did not previously exist. I would expect that the installer looks at all the drivers listed in the registry and simply makes sure that there are updated versions of all of them, without checking first whether they existed prior to installing the service pack. After all, if a driver is listed in the registry, and the operating system is instructed to load it, developers could very easily make the assumption that the driver is present on the computer and actually does load.
Regardless of whether the driver file is there or not, I still have to say that the problem is that the registry key should not exist on an AMD-based computer, regardless of what files are laid down on disk. It is not the presence of a file that causes a problem, but the instruction to load that file on boot, and that instruction is represented by the registry key. It is perfectly legitimate to lay down all kinds of files on disk during installation but not load them. In fact, HP itself lays down the intelppm.sys file in the i386 directory - the on-disk cache directory of operating system files. This strategy is also used successfully by Microsoft Office, Windows Vista, Windows Server 2008, and several Adobe Products. It prevents the user from needing access to the original disks to update, repair, or modify an installation.
What this means is that if you have one of the affected HP desktop computers you can prevent the problem before it even starts. Before installing the service pack go to a command prompt and run either of these commands:
reg add HKLM\System\CurrentControlSet\Services\Intelppm /v Start /d 4 /t REG_DWORD
sc config intelppm start= disabled
Both commands will disable the driver before you install the service pack and will prevent the problem from ever occuring.
Potential impact on Media Center
Two separate posters have reported problems with Windows Media Center after this work-around. At this point, I am not ready to say that this is caused by using the work-around, but if you have this problem, I would appreciate a note to confirm it.
Logically, it could be related. This is pure speculation, but based on what I know about the Digital Rights Management (DRM) in Media Center it may detect the change in hardware, disabling the intelppm driver, as a hostile action and disables viewing DRM protected content. Unfortunately, Comcast Cable puts a DRM signal into some of their cable channels, which means you can no longer watch those channels. You would also be unable to watch previously recorded content. The content provided by Comcast is not actually encrypted, but Windows enforces the DRM nevertheless.
I had a very similar problem with Media Center last year. At the time I was unable to resolve it. However, I would encourage anyone who has this problem to try resetting the DRM components in Media Center. If that does not work, try re-enabling the intelppm driver and see if that helps. It should be safe to do so if the intelppm.sys file is not present in the %systemroot%\system32\drivers directory (check first), and once the computer has booted properly after the service pack installation.
Bill Castner, who is rapidly becoming my new hero, also posted a solid work-around for Media Center problems over in the AumHa forums. Try that one as well, it may solve your problems too.
Second problem, affecting certain AMD motherboards
The second problem type manifests itself in a different error code during boot, and also seems to affect only AMD-based computers. The error code will say something similar to:
Problem was detected and windows has been shut down to protect your computer from damage.
The BIOS in this system is not fully ACPI compliant
You will then get some information about how to update your BIOS. The BIOS is the basic operating system built into the computer that handles reading and writing from disk and memory, as well as some other devices. That is most likely not your problem. The screen ends with the tell-tale error code: STOP: 0x000000A5. If you have that error code, and you just installed SP3, this is most likely your problem.
At the moment, I do not know for sure why this is happening, and I have not personally seen it. The problem appears to be the ASUS A8N32-SLI Deluxe motherboard, also with an AMD processor. Several different AMD processors have been fitted on that board, however, so it seems more likely to be the board than the processor.
The solution is simplicity itself: insert a USB flash drive, or some other form of secondary storage mechanism, before booting the computer. The people have that have seen this problem report that it goes away when they do. The catch is that the computer will only boot with a secondary drive attached. If you remove the secondary drive it will no longer boot.
It also appears that this could be related to using a USB mouse. If you have a USB mouse, try moving it to the PS/2 port instead (the little round port, you should have received an adapter with your mouse). That seems to resolve the problem without the use of an external USB flash drive.
If you have this problem, and either solution helps, or even if they do not help, I'd appreciate a comment on the blog so we can figure out what is going on here.
Other STOP Errors
Every time a service pack is installed, or any major maintenance like it is performed, a certain, very small, number of computers seem to not come back up. The reasons could range from malware on them that is conflicting with the installation or the new files, to bad hardware that somehow failed at that very moment.
For that reason, there may be other STOP errors involved in this problem. Due to the default settings in XP, all of them would result in an endless reboot cycle. Only if there are many of them does it usually indicate a problem with the service pack. A fair number of people are reporting an error code 0x00000024. It usually means either that the file system driver, ntfs.sys, has been corrupted, or you have a hard disk with bad blocks in bad places. It could be totally unrelated to the service pack. At this point, I just do not have enough details to tell. This one seems to be more related to Intel-based computers though.
It is also possible that 0x00000024 has to do with a faulty video driver. I have seen a couple of reports of crashes caused by the ATI Catalyst 8.4 drivers, and one of a crash involving an nVidia driver of some kind, but I do not know which one. To see if that is your problem, try booting into Safe Mode or VGA mode. If VGA mode works you very likely have a video driver issue. Gary Barclay, in a comment below, pointed out that the 8.432 version of the driver may be the one that is faulting, and that version 8.467 appears to work properly. If anyone else can confirm that I'm sure may others will be happy about it.
If you are getting the 0x00000024 error, there are some things to try:
- There is some good information in the Microsoft knowledge base on how to trouble-shoot STOP errors. Try following that.
- If you have multiple drives in the computer, disconnect them one by one and try booting. The problem may not be on your primary drive and this could let you isolate which one has the problem.
- Run chkdsk /r. The problem could be file system related, and chkdsk could fix it. However, to do that you have to boot the computer successfully. If you have a 0x00000024 error, it will not boot even into safe mode. You will need to follow the instructions in the Recovery Console or WinPE sections below to boot the computer.
- Replace the ntfs.sys driver. If the driver file itself has become corrupted there is a backup copy in the %windir%\system32\dllcache folder. If nothing else helps, you could try replacing the version in %windir%\system32\drivers folder with the one from dllcache and see if maybe it was a corrupted file problem.
- If you have an ATI or nVidia driver for for your graphics card, notably the ATI Catalyst 8.4, and your computer will not boot, try booting into VGA mode and see if that works. If it does, you almost certainly have a video driver problem. Uninstall the driver and see if Windows will find a better one. If this works for you, please either contact me using the contact link, or post a comment, so others can learn what is really happening here.
There have also been sporadic reports of video driver problems as well as other issues, like the VPN issues. Most of those have to do with some form of third-party software that does not work with SP3. If you have a problem that is not covered here, it would be good if you could let us know. It may be related to SP3, in which case others may have it too. The VPN issue mentioned by one of the posters has me very interested, for example.
Other people are reporting that the computer is complaining that a particular file is corrupted. Sometimes the corruption results in a blue screen, other times something does not work right after the computer reboots. At this point I am not sure what could be causing this, and I would encourage anyone who runs into that problem to call the Microsoft support line listed above. If they manage to figure out what the problem is, please post back here so the rest of us can find out.
Conflicts with Certain Wireless Card Drivers
Tim Steele read the blog and found that his problem was not solved. After doing some more research he discovered a conflict with certain wireless cards. I asked if I could post his discovery. This is what he wrote:
Some 802.11b wireless cards cause XP to blue screen after installing SP3
If you have any of the following 802.11b wireless cards you'll see a blue screen after installing SP3:
SMC 2635W, Belkin F5D6001, Linksys WPC11 v1, Blitz NetWave Point PC, Xterasys Cardbus XN-2411b, D-Link DWL-520 Revision C, Xterasys Cardbus XN-2411b, Fiberline FL-WL-200X, 3com Office Connect 3CRSHPW796, Corega WLPCIB-11, SMC 2602W V2, and D-Link DWL-520 Revision C.
These cards all use the adm8211 chipset. The driver was provided by ADMtek and badged by the vendors. The last version on the net seems to be 1.80. The D-Link driver is WHQL certified and signed.
There are plenty of adm8211 cards out there inside machines which are about to update to SP3, Windows Update doesn't check whether you have one of these cards before automatically installing SP3, so the effect for many users will be a mysterious blue screen and no obvious cause.
It's not clear whether the vendors or Microsoft should be responsible for fixing this, but surely as a minimum SP3 should not install on machines with this hardware.
Conflicts with Anti-Malware Software
Gregg Keizer wrote an interesting couple of articles in Computer World (second piece is here) about conflicts between Symantec's anti-malware suites and SP3. It appears all but certain that the anti-malware suites cause registry corruption, failures in device manager, and other problems, when you install SP3. An interesting thread on Symantec's support forums documents some of the problems. There are directions for how to disable Symantec's software in another thread.
The security suites add significant hooks into the operating systems. It is quite possible that they will prevent a major installation, such as a service pack, from completing properly. For that reason, you should at the very least disable any anti-malware or security software you have installed prior to installing the service pack. If you can uninstall it, install the service pack, and then reinstall the anti-malware software, you will probably have even greater chance of success.
Using the Recovery Console in XP
If you cannot boot into safe mode you can try using the Recovery Console in Windows XP. This requires you to have a Windows XP CD. Knowledge Base Article 307654 has directions on how to use it. You do not need to follow the instructions for how to install it. In fact, if you have a problem like the 0x00000024 issue above, you probably can not boot from an installed recovery console anyway.
In brief, to boot from the recovery console in XP, do this:
-
Insert your Windows XP CD
-
Boot the computer
-
Select to boot from the CD. On many computers you have to hit a button to do that. On Dell computers the button is usually F12. On HP it is usually ESC.
-
The computer will work for a while and eventually you get a screen that says "Welcome to Setup". Hit the R key here
-
If will ask you which installation you want to boot. If you have several XP installations on this computer, select the one you want. Of course, if you have several installations, and one still works, you would not need these steps.
-
Type the administrator password for the installation you need to repair.
At this point, you should be at a command prompt. The commands you can run are very limited and they are often different from what you are used to. If you have disabled the intelppm driver on an Intel-based computer and need to re-enable it, run "enable intelppm SERVICE_SYSTEM_START".
If you need to run chkdsk you can do it from the recovery console window as well. The C: drive is the boot volume in your Windows XP installation. To run the full check run "chkdsk c: /p /r"
Build a WinPE Disk on a Flash Drive
Another option, recommended for advanced users, is to have a Windows PE disk handy. Windows PE is a miniature version of Window that can boot from a CD, and starting with Windows Vista, a USB Flash Drive. I wrote up directions on how to build a Flash Drive with Windows PE in the Vista book, and there are now also directions on TechNet. You need to have access to a computer that boots, and you need a copy of the Automated Installation Kit (WAIK). Once you burn the AIK image to a disk you can install it and start building your Win PE disk.
Using a Windows PE disk you get access to all the normal tools, like regedit. It has far more features than what you have with the recovery console, but requires a lot more prep work to get started.
Removing SP3
A few people decided the problems were sufficient to just remove SP3 altogether. If you have a problem that is not covered above, that may be your best option for the moment. Microsoft just published an article on how to remove the service pack. It includes information on how to remove it even from the Recovery Console, so even if your computer will not boot you should be able to do it.
