Abstract
Building robust and general reasoning ability is a central goal in the development of large language models (LLMs). Recent efforts increasingly turn to code as a rich training source, given its inherent logical structure and diverse reasoning paradigms such as divide-and-conquer, topological ordering, and enumeration. However, reasoning in code is often expressed implicitly and entangled with syntactic or implementation noise, making direct training on raw code suboptimal. To address this, we introduce TracePile, a large-scale corpus of 2.6 million samples that transforms code execution into explicit, step-by-step chain-of-thought-style rationales, which we call Chain of Execution (CoE). The corpus spans domains including mathematics, classical algorithms, and algorithmic competition, and is enriched with variable-tracing questions and code rewritings to enhance logical granularity and code diversity. We evaluate TracePile using three training setups: continue-pretraining, instruction tuning after pretraining, and two-stage finetuning. Experiments across four base models (LLaMA 3, LLaMA 3.1, Qwen-2.5, and Qwen-2.5 Coder) and 20 benchmarks covering math, code, logic, and algorithms demonstrate consistent improvements. Notably, TracePile boosts LLaMA 3.1-8B by 7.1% on average across nine math datasets and delivers clear gains on LiveCodeBench, CRUX, and MMLU under two-stage fine-tuning.
Introduction
Large Language Models (LLMs) have demonstrated remarkable proficiency in coding, leading researchers to hypothesize that code serves as a "secret sauce" for broader logical reasoning. However, training on raw code presents a significant challenge: the underlying logic is frequently obscured by syntactic requirements and implementation-specific noise. While Chain of Thought (CoT) prompting encourages models to explain their steps in natural language, standard datasets often bypass the "inner life" of the code—the granular state changes that occur during execution.
We argue that if a model cannot accurately trace how a variable changes within a loop or how a recursive call stack unfolds, it has not fully internalized the underlying logic. To bridge this gap, we propose Chain of Execution (CoE), a method that teaches models to "think like a debugger" by narrating the step-by-step execution of code in natural language.
The TracePile Corpus
The core of our contribution is TracePile, a massive corpus comprising 2.6 million query-code pairs totaling approximately 19 billion tokens. TracePile is designed to provide rigorous CoE supervision across three primary domains:
- Algorithmic Competitions: 480k samples sourced from platforms like Codeforces.
- Classical Algorithms: 949k samples based on the CLRS (Introduction to Algorithms) framework.
- Mathematical Solver Code: 1.17M samples derived from OpenMath.
Data Generation and Diversification
To generate high-quality CoE rationales, we utilized frontier models (such as Qwen-2.5-72B-Instruct) to produce detailed, play-by-play commentaries of program execution. To ensure the models learned generalizable logic rather than mere pattern matching, we implemented two diversification strategies:
- Query Diversification: Beyond simple output prediction, we tasked models with tracing specific internal states, such as the contents of a stack after a specific iteration or the value of a variable at a particular line.
- Code Diversification: We rewrote code snippets into multiple functional equivalents—for example, converting recursive functions to iterative ones—to decouple the logic from specific syntactic structures.
Experimental Results
We evaluated TracePile across four model families (LLaMA 3, LLaMA 3.1, Qwen-2.5, and Qwen-2.5 Coder) using three training paradigms: Continue-Pretraining (CPT), Instruction Tuning after CPT, and a Two-stage Finetuning approach.
Mathematical and Algorithmic Mastery
The impact of CoE supervision was most pronounced in structured reasoning tasks. TracePile improved LLaMA 3.1-8B performance by an average of 7.1% across nine mathematical benchmarks. In algorithmic tasks, the gains were even more dramatic; on the GraphWiz benchmark, LLaMA 3-8B's performance surged from a baseline of 4.5% to 46.0% after TracePile training.
General Reasoning and Transfer
Crucially, the reasoning skills acquired through CoE supervision transferred to non-coding domains. In two-stage finetuning experiments, models showed significant improvements on "out-of-domain" tasks like Zebra Logic and MMLU. This suggests that tracing a Depth-First Search (DFS) algorithm, for instance, helps a model better manage constraints and state-tracking in general logic puzzles.
Discussion and Limitations
Our findings indicate that execution traces provide a more logically rigorous rationale than standard human-written explanations. Because code follows strict, deterministic rules, CoE rationales serve as a "gold standard" for step-by-step reasoning. Furthermore, our ablation studies confirmed that supervising the process (CoE) is significantly more effective than supervising only the result (Input/Output pairs).
However, several challenges remain:
- Domain Bias: The dataset is grounded in code, which may limit its utility for commonsense reasoning or non-structured world knowledge.
- Computational Cost: Generating high-quality CoE traces via frontier models is resource-intensive.
- Context Constraints: Complex algorithms often produce execution traces that exceed standard context windows (e.g., 8k tokens), requiring the truncation of the most sophisticated reasoning patterns.
Conclusion
TracePile and the Chain of Execution paradigm represent a significant step toward more robust LLM reasoning. By transforming implicit code logic into explicit execution traces, we provide models with a structured framework for thinking through complex problems. Future work will explore expanding CoE to domains such as legal reasoning and strategic planning, as well as methods for handling longer, more complex execution sequences.
Summary updated