TL;DR
DECAF (DECompilation with Automated Feedback) dramatically improves neural decompilation by combining large language models (LLMs) with automated compiler feedback. Instead of relying on a single model guess, DECAF generates multiple candidate decompilations, filters out those that fail to compile, and uses a specialized neural reranker to select the most functionally accurate and idiomatic code. This approach skyrockets the functional correctness rate on the ExeBench benchmark from a baseline of 26.0% to an impressive 83.9%.
What Problem Does This Paper Address?
Decompilation—translating low-level machine assembly back into human-readable source code—is a cornerstone of reverse engineering, malware analysis, and software security. Traditional decompilers (like Ghidra or Hex-Rays) are structurally accurate but produce ugly, non-idiomatic code stripped of meaningful variable names, types, and comments.
Recently, generative AI and LLMs have been applied to this problem. While LLMs excel at producing highly readable, natural-looking code, they suffer from a fatal flaw: hallucination. They frequently generate code that either fails to compile or subtly alters the program's logic—for instance, omitting a critical loop-break condition (as illustrated in Figure 1).
The authors address a fundamental question: How can we leverage the readability of neural network outputs while guaranteeing functional correctness?
Methods
Instead of training ever-larger models on more data, DECAF introduces a multi-step "generate-filter-verify" pipeline that uses the compiler itself as a feedback mechanism.
- Traditional Initialization: The target binary is first processed through a traditional decompiler (Ghidra) to extract a structural baseline.
- Diverse Generation: This baseline is fed into a custom 22-billion-parameter LLM (
Decaf-Gen-22b). Using temperature sampling, the model generates a diverse set of $N$ (e.g., 32) candidate source code variations. - Compilation Filtering: DECAF attempts to re-compile every single candidate using the reference compiler (
gcc). Any candidate that throws a syntax or compilation error is immediately discarded. - Neural Reranking: For the remaining valid candidates, DECAF compiles them to assembly and uses a second, 32-billion-parameter LLM (
Decaf-ReRanker-32b) to score how semantically close the candidate's assembly is to the original target binary's assembly. If an exact byte-wise match is found, it is prioritized; otherwise, the neural reranker selects the highest-scoring candidate.
To keep the assembly comparison within the LLM's context window, the reranker evaluates a diff between the target disassembly and the generated candidate's disassembly (see Figure 2).
Datasets
The authors trained and evaluated their models using two primary datasets:
- ExeBench: A large-scale dataset consisting of standalone, compilable C functions mined from open-source projects. The authors utilized the
REALandSYNTHsplits, compiling them at both-O0and-O2optimization levels for x86-64 architectures. - NIST Juliet Test Suite (v1.3): A benchmark containing documented software weaknesses (CWEs). This was used to evaluate DECAF's downstream utility in recovering vulnerabilities from stripped binaries so they can be flagged by static analysis tools like CodeQL.
Key Insights and Ideas
- The Power of Multiple Guesses: A single LLM pass has a limited chance of being completely correct. However, sampling 32 candidates yields an 88% chance that at least one candidate is functionally perfect (Figure 4). The core challenge isn't generation; it's selection.
- Correctness Breeds Readability: One might worry that optimizing strictly for compiled byte similarity would force the LLM to output ugly, compiler-like code. Surprisingly, the authors found the opposite: reranking for functional correctness actually improved the source code similarity (Edit Distance) to the original human-written code.
- Combating "Shortcut Learning": LLMs evaluating unstripped binaries often cheat by relying heavily on function names rather than understanding the underlying logic. Training and testing on stripped binaries forced the models to focus purely on semantics.
Key Results
DECAF achieves state-of-the-art performance across the board on ExeBench:
- Functional Correctness: On the stripped
REAL -O2split, DECAF achieved an 83.9% functional solve rate, vastly outperforming the prior state-of-the-art model, IDIOMS, which sat at 26.0%. - Exact Byte-Wise Matching: DECAF achieved a 70.9% exact byte-wise match to the original compiled binary.
- Vulnerability Recovery: When tested on the Juliet dataset, DECAF achieved an F1 score of 29.8% for vulnerability detection via CodeQL, compared to a meager 1.7% achieved by raw Ghidra output.
Limitations and Future Work
Despite its strength, DECAF has a few notable limitations:
- Compiler Dependency: The neural reranker is sensitive to the compiler used. When the authors stress-tested the GCC-trained reranker against code compiled with
Clang, performance dropped significantly due to distribution shift in the generated assembly patterns. - Scope Constraints: The current implementation is restricted to the C programming language and the x86-64 architecture. It does not yet account for alternative architectures (like ARM or RISC-V) or binaries intentionally obscured by code obfuscation techniques.
Future work will focus on training rerankers on highly diverse compiler configurations to make the system robust against unknown binary origins.
Summary updated