x86 Stack Conventions

In an attempt to get a more solid understanding of assembly for reverse engineering and exploit development I started watching the OpenSecurity Introductory Intel x86: Architecture, Assembly, Applications, & Alliteration training videos found here. Day 1 Part 1 had a great explanation of how programs interact with the stack when functions are called. I just wanted to write up a summary here to solidify my understanding and have an easy reference for later.

First it’s worth mentioning that these are recommended conventions and not required. Different compiler options can cause this behavior to be different, or non-existent.

The first and most common standard for compiled c code is cdecl (c decleration). In this standard the following actions are performed when calling a function. All of the memory addresses in this example are just random numbers.

1. First the program pushes any register values that it doesn’t want to lose to the stack. For example maybe it has something really important stored in EBX. When the new function is called it could overwrite EBX with something else. So the following command is executed so EBX can be restored after the called function completes. S

Push EBX 
(Because the stack grows from high memory addresses to low a push subtracts 4 from ESP)

2. Next the memory addresses of the parameters for the called function are pushed onto the stack from right to left. For example in c let’s say you are going to call this function(param1, param2). Let’s say the value of param1 is stored at memory address 0x00000001 and the value of param2 is stored at 0x000000F0. You would see the following assembly commands.

Push 0x000000F0
Push 0x00000001

3. Now the function is called effectively changing EIP into the new function which starts executing it. Call also pushes the next memory address onto the stack. This pushed address is the return address that the called function will use when it’s completed.

Call 0x402039e5 (effectively push next address, jmp 0x02039e5)

4. The first thing that the called function should do is push EBP onto the stack. The EBP register is used to track the base of the stack frame. Since the program is about to start a new stack frame for the new function it needs to save the old EBP value so when the function completes it can rebuild the callers stack frame. Now that the old stack frame info is saved it moves ESP into EBP to create its own empty stack frame.

Push EBP
Mov EBP ESP

5. Now the called function pushes any register values to the stack that it wants to preserve. Maybe it cares about the callers EBX value because it was passed as a reference but it needs to use EBX for other logic.

Push EBX

6. The called function pushes addresses to memory space for its local variables since it knows how many there will be and their types

Push {some addresses}

7. The called function runs its logic. When it needs to use the parameters that the caller passed it does not pop them off the stack (it couldn’t even if it wanted to since they are not on top). It just references their addresses.

8. When the called function is done running typically it will save the address of its return value in EAX. For example if the function returns a string “hello world”, and the string “hello world” is stored at address 0xfefefe34 it will move that address to EAX. It’s also possible to move the actual value to EAX. If the function wants to return the value of 0, Mov EAV 0x0000000.

Mov EAX 0xfefefe34

9. Now that the called function is finished it pops the callers EBP back off the stack.

Pop EBP (since the stack grows from high memory addresses to low a pop adds 4 to ESP)

10. To pass control back to the caller function the called function executes return. Return pops the return address (which should be on the top of the stack now) into EIP

Ret

11. The calling function now decrements ESP so the called functions parameters are effectively no longer on the stack. Technically they are still in memory at the same place but now that they are not in between EBP and ESP and they will never be used and will be overwritten.

Here is a super simple example from the OpenSecurity PowerPoint slides.

SimpleCdecl

Another popular standard for is stdcall which is typically used by Microsoft for c++. It works the same as cdecl with the following change. On step 10 instead of executing ret then allowing the caller to clean up the parameters the called function will execute ret followed by a parameter to decrement ESP. This effectively does step 10 and 11 at the same time.

Ret 0x00000008 (return and subtract 8 from ESP)

Leave a Reply

Your email address will not be published. Required fields are marked *