Data Execution Prevention in WindowsXP Service Pack 2

Starting with Windows XP Service Pack 2, Windows XP supports a feature called Data Execution Prevention (DEP) that prevents programs from replacing the original, intended machine instructions in memory with new instructions that could perform malicious acts. This feature became necessary when virus writers and hackers began exploiting bugs in software that can result in malicious program instructions sent from the attacker being written in memory that was supposed to hold just program data. These are often referred to as buffer overrun exploits or stack overflow exploits. If the program's path through memory eventually takes it into the malicious code, the injected instructions can do anything you can do; that is, the program has your privileges, can access any file you can, can infect other programs, and so on.

Data Execution Prevention uses two different mechanisms to guard against this type of attack: First, it uses Windows software mechanisms to prevent programs from writing any new instructions into sections of memory that were originally designated as holding instructions. Second, it prevents programs from executing instructions from any section of memory that was originally designated as holding only data. The second form of protection is the stronger of the two, but it is only available with some CPU chips, including all 64-bit processors from AMD and Intel, Intel's Pentium D and Pentium 840 Extreme Edition processors, and AMD's Sempron processors. This second mechanism is called Execute Disable or ED by Intel, and No Execute or NX by AMD, and it's used when available whenever Data Execution Prevention is enabled in Windows.

By default, when Windows XP Service Pack 2 is installed, DEP is enabled only for Windows components themselves. To protect all applications, right-click My Computer and select Properties to open the System Properties dialog. Select the Advanced tab, and click the top Settings button under Performance. Select the Data Execution Prevention tab

To enable DEP for all applications, select Turn On DEP for All Programs and Services Except Those I Select.

If you change Data Execution Prevention settings, you'll need to restart Windows. When enabled for all applications, you may find that an application that used to work suddenly fails with a dialog box that says "Data Execution PreventionA Windows security feature has detected a problem and closed this program." In this case, you should contact the manufacturer's tech support to see whether this is a known issue, or if an update is available.

If you determine that the application is actually safe but just happens to require the ability to write modified instructions in order to work, you can instruct Windows to disable DEP for this application. Back in the DEP setup dialog click Add, and then browse to select the .EXE file that corresponds to the application in question. Click OK to save it in the list of exceptions.

If Windows halts with a blue screen, or reboots repeatedly when you restart it after enabling Hardware DEP (or after updating a device driver when Hardware DEP is enabled), one of your device drivers is executing code from "No Execute" memory and terminating. Use one of the following methods to disable DEP. First, try to boot Windows in Safe mode:

If Windows won't even boot in Safe mode, you'll need to take the more drastic step of manually editing the boot.ini file on the hard drive that contains Windows. To do this, follow these steps:

e:  attrib -r -h -s boot.ini  notepad boot.ini  

attrib +r +h +s boot.ini  

When Windows boots successfully, log on as a Computer Administrator and check the Event Log for an indication of which driver failed during startup. Update it or roll it back before enabling hardware DEP again.