Preventing Buffer Overflow Exploits, Part 2

by Ibrahim Haddad

The buffer overflow exploit has become almost ubiquitous in recent years. This article looks at how DSM guards against such exploits when implemented as a Linux module.

The troubling  thing about buffer overflow exploits is that good programming practices could wipe out even potential exploits; however, that simply has not happened. The defense against such exploits should revolve around controlling access to sensitive systems, installing software updates that replace exploitable software, and being aware of what a buffer overflow exploit looks like when your system is the intended victim.

Since buffer overflow exploits are one of the most popular attacks, there were many ways to try to prevent them.

  1. The first solution is to write correct code where the programmer checks the size of the buffer; for instance, using strncpy instead of strcpy. In Part 1's Listing 3 (vulnerable.c), the strncpy function should be used instead the strcpy function. However, since one cannot expect that the code to be always written correctly, there is a need for system solutions.
  2. Buffer overflow exploits depend on an executable stack. One obvious solution is to make the stack segment non-executable; such kernel patches are available for Linux. This approach prevents buffer overflow exploits; however, it will create problems in the signal handling procedures. Signal handler returns in Linux require an executable stack. Furthermore, signal handlers are crucial in an operating system; thus, a temporary executable stack for signal handlers must be implemented. By removing stack execution from the system kernel, buffer overflow exploits can be prevented. However, this approach suffers from the code being non-portable, and operating system handling behavior is modified and may be unpredictable. Another approach is the PaX project. The goal of the PaX project is to research various defense mechanisms against the exploitation of software bugs that give an attacker arbitrary read/write access to the attacked task's address space. This class of bugs contains among others various forms of buffer overflow bugs (be they stack or heap based), user supplied format string bugs, and so on.
  3. Buffer overflow exploits are possible on processors on which the stack grows down. By moving more of the data then a function’s local buffer can store, a return address can be overwritten. This situation cannot be achieved on the processors with the stack growing up because the return address has the lower address than the local buffer. Therefore, the strcpy function moves data to higher addresses and never reaches the return address. It can happen that some other part of the stack can be overwritten but not the return address. Therefore, it is impossible to jump to our shell code. Another solution would be to make the stack gown up.
  4. Other solutions are implemented in compilers. A random extra field is placed on the stack when the function is called. After exiting from call, this field is checked for corruption. This solution can be used at compile time. However, we must recompile all existing programs, and problem arise when dealing with binaries and source code that is not available.
  5. The solutions to be discussed in this article are implemented in the kernel and are based on the mandatory security and Linux Security Module (LSM) hooks.
    1. One solution is offered by the National Security Agency (NSA) and is called the Security Enhanced Linux (SE Linux)
    2. The other solution is offered by Ericsson Research Open Systems Lab and is called the Distributed Security Module (DSM). At the time of writing, the stable version is DSI 0.3. DSI 0.4 is the beta version with ongoing development work.

    Both solutions are similar in principle because they are based on the same hooks in the Linux kernel. However, they differ in the actual implementation, and the security policies, performance, and usability. We did not try to configure SE Linux for the purpose of this exercise since the policy setting is a bit more complex.

    For the purposes of this article, we will describe how the DSM module guards against buffer overflow exploits.

DSM and Buffer Overflow Exploits

The mandatory access control implemented in DSM is not limited to the traditional two levels of security: root and user. Rather, the security policy, which is independent from the implementation, decides the access rights of the executing process. DSM is based on the LSM infrastructure. The LSM framework does not provide any extra security in the Linux kernel. Rather, it provides the infrastructure with support for the development of security modules. The LSM kernel patch adds security fields to kernel data structures and inserts calls (called hooks) at special points in the kernel code to perform  module-specific access control checks.

We will come back to our example in Part 1's Listing 4 where we tried to spawn the shell with root privileges. To do this, the buffer overflow program must call the system call execve.

When the vulnerable process performs the execve system call, the hook will pass the control to DSM. DSM, based on the loaded policy, will grant or reject the access. In our case, we want to prove that DSM will stop executing the cracked program.

So how do we accomplish this?

>> DSM Preventing Buffer Overflow Exploits

This article was originally published on LinuxPlanet.

DSM Preventing Buffer Overflow Exploits

In this section, we demonstrate how the DSM module prevents buffer overflow exploits.

Step 1: First, we extend the security of the system by loading the DSM module as root:

$/sbin/insmod lsm.o # for historical reason, the name of the module is lsm

Step 2: Next, we update and load the security policy (we explain the directives later on in the text):

$vi policy_file

Add the following line (if not present):

      1 2 128 0
$./UpdatePolicy policy_file

As of DSI 0.3, this has been much improved and instead of using a plain text file to express the security policy, it is now done with XML. Therefore, if you experiment with DSI 0.3 or 0.4 then you will express the policy in XML.

Step 3: Then, we change the security ID of the vulnerable program to 1:

$SetSID vulnerable 1

Step 4: Now, we are ready to repeat the same example of the buffer overflow exploit:

$ ./vulnerable $RET
$Error ./vulnerable: not found

Did we gain root privileges? Let us check:


No! We are still a normal user.

Step 5: Exit from the shell spanned by the buffer overflow exploit:


When executing the second ./vulnerable $RET an error is returned (the error code must be changed to reflect what really happened here) and the system will not allow to start a new shell with root privileges.

To be sure that our module is the source of this error, you can unload the DSM module and run the vulnerable program (Step 4) once again. After the DSM module is unloaded, there should be no error message and the shell will be started with root privileges same as before.

Here is the explanation of what really happened.

In Step 2, we updated the policy by adding the line (1 2 128 0). There are four fields: source security ID (1), target security ID (2), class (128) and permission (0). This means that the subjects with the security ID = 1 cannot perform the operation in class = 128 on the objects with the security ID = 2. We implement the class 128 called DSI_CLASS_TRANSITION in this manner and it covers the ELF executable checking at the load time.

In Step 3, we assigned the security id 1 to the vulnerable program. By default, all the subjects start with the security id 2 when it is not explicitly stated otherwise in the security policy file. The third parameter in the policy record is 128. This value describes the operation (look at the source code). In our case, it means the operation of spanning a process. Therefore, when the vulnerable program with the security id 1 executes the code stored in the vulnerable buffer, at one point the system call (0x80) is performed. The system call passes the execution to the kernel and later to the security hook embedded in the kernel. In the security module (DSM) the security id of the subject (vulnerable) is validated. Based on the policy the operation of spanning will not be allowed.

Now to secure the kernel, we have to find all the vulnerable processes and assign them the security ids that do not allow spanning of other processes. Please note that there is no need to have the source code of the vulnerable programs. Based on the process creation tests, which were presented in a previous article that was published in the Linux Journal, we can safely assume that the overhead of the added operations is minimal and will stay between 1 percent and 2 percent.

>> Conclusion

This article was originally published on LinuxPlanet.

Wrapping Up

Buffer overflow exploits are one of the most interesting security vulnerabilities and are used in a majority of security attacks against Linux and Unix-like operating systems. DSM guards against such exploits, and it is implemented as a Linux module. DSM also provides many other features, such as transparently controlling the access in the distributed environment of Linux clustered servers.

Some notes that should be taken into consideration as well:

  • There have been a lot of changes in the implementation of DSI from 0.1 to 0.2, then to 0.3, and now to the unstable 0.4. Therefore, please make sure to read the documentation provided with DSI to ensure that what you are trying to experiment is valid for the version of DSI that you are using.
  • To get more information, please visit DSI/DigSig Project and subscribe to the DSI mailing list.

In Parts 1 and 2 of this article, we presented and demonstrated that mandatory access control implemented in DSM can prevent against buffer overflow exploits. The security mechanisms were implemented in different levels of the executing system. Because many existing applications are vulnerable to the buffer overflow exploits, one of our goals with DSM was to make the security transparent to the applications, so even the existing programs can be secured without any modifications. The DSM source code is provided as open source and is available for download from the DSI Web site.

This article was originally published on LinuxPlanet.

This article was originally published on Thursday Jan 26th 2006
Mobile Site | Full Site