Binary Exploitation: From Buffer Overflows to Modern Techniques

Part of the Cybersecurity Skills Guide — This article is one deep-dive in our complete guide series.

By HADESS Team | February 28, 2026 | Updated: February 28, 2026 | 5 min read

Binary exploitation is the practice of finding and abusing memory corruption vulnerabilities in compiled software. It requires understanding how programs actually execute at the CPU and memory level — how the stack and heap are structured, how function calls work, and how modern operating systems try to prevent exploitation.

Buffer Overflows

A classic stack-based buffer overflow happens when a program writes more data to a stack buffer than it can hold, overwriting adjacent memory including the saved return address. By controlling the return address, you redirect execution to your own code.

The basic technique: find a vulnerable function (anything using strcpy, gets, or sprintf without bounds checking), determine the offset from the buffer start to the saved return address, and overwrite it with the address of your payload.

On modern systems, this raw technique is dead. Stack canaries detect overwrites before the function returns. DEP/NX marks the stack as non-executable so you cannot run shellcode placed there. ASLR randomizes memory layout so you cannot predict addresses. Exploitation today means defeating these mitigations.

Return-Oriented Programming (ROP)

ROP is the standard answer to DEP/NX. Instead of injecting new code, you chain together short instruction sequences (“gadgets”) that already exist in the program’s executable memory. Each gadget ends with a ret instruction, so the chain executes by manipulating the stack.

Finding gadgets: use tools like ROPgadget or ropper to scan the binary and loaded libraries for useful sequences. You need gadgets that set up register values, make system calls, or call useful functions.

A basic Linux ROP chain to spawn a shell: set rax to the execve syscall number (59), rdi to a pointer to “/bin/sh”, rsi and rdx to NULL, then execute the syscall instruction. Each register setup might require one or more gadgets.

On 64-bit systems, the calling convention passes arguments in registers rather than on the stack, which means you need pop rdi; ret gadgets to load function arguments.

Heap Exploitation

Heap exploitation targets dynamically allocated memory. The techniques depend heavily on the allocator implementation (glibc malloc, Windows heap manager, jemalloc).

Use-After-Free (UAF): a program frees a heap object but continues to use a pointer to it. If you can allocate a new object of the same size in the freed slot, the dangling pointer now references your controlled data. This is the most common heap vulnerability class in modern software.

Heap overflow: similar to stack overflow but on the heap. Overwrite heap metadata or adjacent object data. In glibc, corrupting chunk metadata can lead to arbitrary write primitives when the allocator processes the corrupted chunks.

Tcache poisoning (glibc 2.26+): the thread-local caching mechanism has simpler safety checks than the main allocator. Corrupting tcache free list pointers can give you arbitrary address allocation.

Format String Vulnerabilities

Format string bugs occur when user input is passed directly as the format argument to printf or similar functions. The attacker controls the format specifiers, which lets them:

• Read memory: %x or %p leak stack values
• Write memory: %n writes the number of characters printed so far to an address on the stack
• Arbitrary read: %s dereferences a pointer from the stack and prints the string

The %n write primitive, combined with careful padding to control the character count, gives you an arbitrary write. From there, you can overwrite GOT entries, return addresses, or function pointers.

Modern compilers warn about format string issues and many codebases have eliminated them, but they still appear in embedded systems, legacy code, and custom protocol parsers.

Next Steps

• Evaluate your exploitation skills with the skills assessment
• Explore reverse engineering and related topics in the skills library
• Check the salary calculator to understand compensation for exploit development roles

Related Guides in This Series

• Low-Level Exploitation: Kernel, Driver, and Firmware Attacks
• OWASP ZAP: Web Application Security Testing
• Penetration Testing Methodology: Scoping Through Reporting

Take the Next Step

Browse 80+ skills on HADESS. Go to the browse 80+ skills on hadess on HADESS.

See your certification roadmap. Check out the see your certification roadmap.

Get started free — Create your HADESS account and access all career tools.

Frequently Asked Questions

How long does it take to learn this skill?

Most practitioners build working proficiency in 4-8 weeks of dedicated study with hands-on practice. Mastery takes longer and comes primarily through on-the-job experience.

Do I need certifications for this skill?

Certifications validate your knowledge to employers but are not strictly required. Hands-on experience and portfolio projects often carry more weight in technical interviews. Check the certification roadmap for relevant options.

What career paths use this skill?

Explore the career path explorer to see which roles require this skill and how it fits into different cybersecurity specializations.

—

HADESS Team consists of cybersecurity practitioners, hiring managers, and career strategists who have collectively spent 50+ years in the field.