TL;DR
The paper introduces Trajectory-Augmented Policy Optimization (TAPO), a framework that advances LLM self-distillation by teaching models to explicitly learn from their own mistakes. Instead of forcing a model to implicitly mimic a perfect distribution, TAPO constructs "micro-reflective" training trajectories that preserve a model's incorrect reasoning up to the point of failure, insert a natural-language diagnosis, and correct the trajectory. This approach significantly boosts mathematical reasoning performance on competitive benchmarks without adding any computational overhead at inference time.
What Problem Does This Paper Address?
Reinforcement learning with verifiable rewards (RLVR) and self-distillation have drastically improved complex reasoning in Large Language Models (LLMs). However, current self-distillation methods rely on implicit distributional alignment. They optimize the model to minimize the KL divergence between its output distribution and a privileged target distribution derived from correct solutions.
The authors identify two fundamental flaws in this paradigm:
- Implicit and uncontrollable supervision: The target distribution acts as a black box. The model learns what a correct distribution looks like but never explicitly practices diagnosing why a reasoning step went wrong or how to fix it.
- Errors are suppressed rather than leveraged: When a model makes a mistake, standard objectives penalize the error and force it back toward a correct path. This prevents the model from learning how to recover from an error state once it's in one, and it compresses predictive diversity during exploration.
To address this, the paper looks to educational psychology—specifically Vygotsky’s Zone of Proximal Development (ZPD)—arguing that effective learning must be anchored in the learner's current understanding, including their mistakes, providing an explicit pathway to correctness.
Methods
TAPO transitions self-distillation from implicit logit alignment to explicit trajectory construction. The pipeline consists of several key innovations integrated into a standard Group Relative Policy Optimization (GRPO) framework:
- Cold-Start Phase: Before RL training, the model undergoes a brief supervised fine-tuning phase (using ~45,000 examples) to equip it with basic error-analysis capabilities and teach it to follow structured XML tags (
<analysis>and<reconstruction>). - Difficulty-Aware Candidate Selection (DCS): During RL, TAPO filters queries to target the model's ZPD—problems where the model generates a mix of both correct and incorrect rollouts.
- Micro-Reflective Trajectory Construction: For eligible ZPD problems, TAPO pairs an incorrect rollout with a correct reference rollout. The actor model itself then generates a new trajectory that copies the incorrect reasoning verbatim up to the first critical mistake, inserts a natural-language reflection/correction phrase, and completes the problem with correct reasoning.
- Decoupled Advantage Estimation (DAE): Merging these new trajectories into the original GRPO sampling group would artificially inflate the group reward mean, leading to "advantage contamination" (which causes response length and policy entropy collapse). TAPO isolates the original rollouts and the constructed reflection trajectories into independent groups to calculate advantages separately.
- OOD Token Suppression (OTS): Because the corrective transitions can introduce out-of-distribution (OOD) tokens, TAPO applies a token-level weighting mechanism based on Distribution Discriminant Theory to down-weight meta-cognitive phrases and preserve the core mathematical learning signal.
Datasets
- Training: The model is trained on a filtered subset of the DeepScaleR dataset, which compiles ~40,000 unique math problems from sources like AIME (1984-2023), AMC, Omni-Math, and STILL. Filtering for the ZPD boundary yields roughly 16,000 training queries.
- Evaluation: The framework is evaluated on three highly challenging mathematical competition benchmarks: AIME 2024, AIME 2025, and HMMT 2025 (Harvard-MIT Mathematics Tournament).
Key Insights and Ideas
- Prefix Preservation is Critical: Replacing an entire failed rollout with a completely new correct path (full reconstruction) performs significantly worse than preserving the exact erroneous prefix up to the mistake. Building on top of partial understanding is far more learnable for the model.
- Error Correction Improves First-Pass Accuracy: Surprisingly, training on error-correction trajectories dramatically improves the model's Direct Solution Rate (DSR)—the rate at which it solves a problem correctly on the first try without any reflection. By internalizing common error patterns, the model learns to avoid them entirely.
- Zero Inference Cost: Micro-reflective trajectories are strictly a training-time intervention. At inference time, the model operates in standard single-pass generation mode without explicit reflection prompts or multi-turn decoding. The error-correction capability is fully internalized into general reasoning behavior.
Key Results
Under the exact same number of training steps, TAPO consistently outperforms both standard GRPO and On-Policy Self-Distillation (OPSD) across all benchmarks when initialized from the cold-start checkpoint:
- AIME 2024: TAPO achieves 62.50% Pass@1, outperforming GRPO (52.92%) by 9.58 percentage points and OPSD (57.71%) by 4.79 percentage points.
- AIME 2025: TAPO matches GRPO at 46.88% Pass@1 but beats OPSD (43.33%).
- HMMT 2025 (Hardest Benchmark): TAPO reaches 31.46% Pass@1, a 2.71 point improvement over GRPO (28.75%) and a massive 7.29 point gain over OPSD (24.17%).
Limitations and Future Work
- Dependency on Cold-Start: Without the initial cold-start phase to pre-align the model to the trajectory format, TAPO's performance degrades on harder benchmarks because the OTS mechanism ends up suppressing too much of the unaligned corrective signal.
- Verifiable Reward Constraint: TAPO currently relies on tasks with clear, verifiable binary outcomes (like mathematics). Extending this explicit trajectory construction to open-ended text generation or code synthesis where rewards are harder to verify remains an open challenge.
Summary updated