Analyzing Chain of Thought (CoT) Approaches in Control Flow Code Deobfuscation Tasks

Seyedreza Mohseni, Sarvesh Baskar, Edward Raff, Manas Gaur

TL;DR

By guiding Large Language Models (LLMs) through explicit, step-by-step reasoning—known as Chain-of-Thought (CoT) prompting—researchers have successfully automated the deobfuscation of complex control-flow modifications in C and LLVM-IR code. This approach significantly outperforms traditional zero-shot prompting, with advanced reasoning models like GPT5 achieving up to a 16% boost in structural recovery and a 20.5% improvement in preserving program semantics.

What Problem Does This Problem Address?

Code obfuscation is a double-edged sword: while it protects intellectual property from theft, it is also heavily utilized by malware authors to hide malicious logic from security analysts. Traditional reverse engineering tools (like IDA Pro or Ghidra) and automated formal methods (like SAT/SMT solvers used in symbolic execution) struggle against advanced obfuscation techniques. Specifically, techniques like Control Flow Flattening (CFF)—which destroys the natural loop and branch structures of a program by routing execution through a centralized switch-based dispatcher—and Opaque Predicates—which add fake, mathematically invariant branching paths—cause severe "path explosion" that can slow down automated solvers by orders of magnitude.

While LLMs have shown promise in understanding high-level code, their ability to untangle low-level obfuscated code (such as LLVM Intermediate Representation) remains largely underexplored. Furthermore, fine-tuning LLMs on obfuscated code is computationally expensive and often teaches the model superficial pattern recognition rather than genuine semantic reasoning.

Methods

Instead of retraining or fine-tuning models, the authors propose a structured Chain-of-Thought (CoT) prompting framework that mimics the manual, step-by-step workflow of a human reverse engineer. The framework operates in five distinct phases:

  1. Obfuscation Detection: The model parses the Abstract Syntax Tree (AST) to identify dispatcher constructs (CFF indicators) and algebraic invariants (opaque predicates).
  2. Control Flow Flattening Recovery: The model extracts individual case blocks from the switch statement and maps out the state transitions.
  3. Control Flow Reconstruction: By identifying loop back-edges and performing a topological sort, the model determines the original execution order.
  4. Opaque Predicate Elimination: The model evaluates the mathematical invariants of conditional branches (e.g., recognizing that $x^2 \ge 0$ is always true) to prune dead, unreachable code blocks.
  5. Code Cleanup: Unused variables (like the dispatcher state variable) are removed, and the AST is normalized into clean, readable source code.

The authors evaluated five state-of-the-art reasoning models using this pipeline: GPT5, OpenAI's o3, Qwen-3 Max, QwQ-32B, and DeepSeek-V2.

An example of an opaque predicate in LLVM-IR where global variable analysis reveals an invariant condition.

Datasets

The approach was validated on a fixed pool of 12 standard C benchmark programs spanning various control-flow complexities. The benchmarks were split into two categories:

  • Group 1 (Lower Complexity): Merge Sort, Heap Sort, Quick Sort, Binary Search, Dijkstra, BFS, DFS, Knapsack, and Matrix Multiplication.
  • Group 2 (Higher Complexity): N-Queens (with solution printing), an AVL tree (with rotations), and Huffman encoding/decoding.

Each benchmark was obfuscated using two widely recognized open-source tools: Tigress and O-LLVM. To ensure the models relied purely on structural logic rather than semantic hints, all comments and algorithm-identifying names were stripped prior to prompting.

Key Insights and Ideas

  • Deobfuscation as Variable Provenance: The authors frame control-flow deobfuscation fundamentally as a problem of tracking variable origins and consumption points. If an LLM can trace how global variables dictate dispatcher states, it can successfully separate genuine program logic from obfuscation artifacts.
  • The "Intrinsic Complexity" Tax: An LLM's ability to deobfuscate code depends heavily on the baseline complexity of the original program. Even with identical levels of obfuscation, more complex algorithms (Group 2) caused a much steeper decline in model performance than simpler ones (Group 1).
  • Taxonomy of LLM Deobfuscation Failures: The paper categorizes LLM errors under heavy obfuscation into three distinct types:
  • Simple Omissions: The control flow is correct, but the model forgets boilerplate code like #include headers.
  • Structural Hallucinations: The code is syntactically complete, but key data structures are malformed.
  • Semantic Hallucinations: The most dangerous type, where the model outputs clean, compilable code that completely alters the program's original behavior (e.g., substituting a maximum function with an addition function).

Key Results

The performance was measured using two metrics: Structural Similarity Score (SRS) for control flow graph accuracy, and BLEU score for semantic output preservation.

  • GPT5 emerged as the top performer, achieving near-perfect reconstruction across multiple benchmarks. With CoT prompting, it achieved an average gain of 16% in SRS and 20.5% in BLEU compared to zero-shot prompting.
  • Reasoning models significantly outperformed standard LLMs. Models operating in explicit reasoning modes proved far more capable of maintaining state across long sequences of basic blocks.
  • Tigress proved more challenging than O-LLVM. O-LLVM's opaque predicates feature less structural variation, making them easier for the models to solve, whereas Tigress introduces greater structural diversity that led to more frequent model errors.
Overall average BLEU and SRS scores across the evaluated models, highlighting GPT5's strong performance.

Limitations and Future Work

While the results are promising, the study is limited to two open-source obfuscators (Tigress and O-LLVM) and focuses primarily on control-flow obfuscation rather than data obfuscation. Additionally, as the density of opaque predicates increases, less powerful models degrade rapidly, sometimes failing to produce compilable code entirely.

Future work aims to expand this methodology to more advanced obfuscation techniques, explore more sophisticated prompting strategies like Tree-of-Thought (ToT) to map out alternative execution paths, and evaluate the pipeline against real-world malware families such as Emotet.

Summary updated

Abstract

Code deobfuscation is the task of recovering a readable version of a program while preserving its original behavior. In practice, this often requires days or even months of manual work with complex and expensive analysis tools. In this paper, we explore an alternative approach based on Chain-of-Thought (CoT) prompting, where a large language model is guided through explicit, step-by-step reasoning tailored for code analysis. We focus on control flow obfuscation, including Control Flow Flattening (CFF), Opaque Predicates, and their combination, and we measure both structural recovery of the control flow graph and preservation of program semantics. We evaluate five state-of-the-art large language models and show that CoT prompting significantly improves deobfuscation quality compared with simple prompting. We validate our approach on a diverse set of standard C benchmarks and report results using both structural metrics for control flow graphs and semantic metrics based on output similarity. Among the tested models and by applying CoT, GPT5 achieves the strongest overall performance, with an average gain of about 16% in control-flow graph reconstruction and about 20.5% in semantic preservation across our benchmarks compared to zero-shot prompting. Our results also show that model performance depends not only on the obfuscation level and the chosen obfuscator but also on the intrinsic complexity of the original control flow graph. Collectively, these findings suggest that CoT-guided large language models can serve as effective assistants for code deobfuscation, providing improved code explainability, more faithful control flow graph reconstruction, and better preservation of program behavior while potentially reducing the manual effort needed for reverse engineering.
BibTeX
@inproceedings{mohseni2026analyzing,
  author = {Seyedreza Mohseni and Sarvesh Baskar and Edward Raff and Manas Gaur},
  title = {{Analyzing Chain of Thought (CoT) Approaches in Control Flow Code Deobfuscation Tasks}},
  booktitle = {EuroS\&P (Workshops)},
  year = {2026},
  doi = {10.1109/eurospw72509.2026.00033},
}

Loading related content...

Figure