TL;DR
Instead of training LLMs to imitate entire reasoning "trajectories" or rewarding them for final answers, this paper proposes edit-based training. By identifying the exact point where a model first violates a reasoning instruction and providing a minimal textual correction (often just a token or two), researchers can steer model behavior with surgical precision while preserving its original problem-solving capabilities.
What Problem Does This Paper Address?
As we move toward "System 2" models that think before they speak, we need more control over how they reason. A user might want a model to "reason in Chinese but answer in English" or "check every assumption before proceeding."
Currently, we have two main ways to train this:
- Supervised Fine-Tuning (SFT): We give the model a full "gold standard" reasoning trace to copy. The problem? The model often imitates incidental quirks of the teacher model rather than just the logic, leading to "entropy collapse" where the model loses its own diverse ways of solving problems.
- Reinforcement Learning (RL): We give the model a reward based on the final outcome. The problem? This has poor credit assignment. If a model writes a 500-word reasoning trace and fails because of one tiny mistake in the middle, RL penalizes the whole thing, even the parts that were actually good.
The authors seek a middle ground: textual supervision that is sparse, localized, and focuses only on the specific failure point.
Methods
The core of the paper is the edit-based training procedure. It works like a surgical intervention:
- Sample: The model generates a reasoning trace.
- Identify: An "Editor" (which can be a script or a stronger LLM like GPT-5) scans the trace to find the first point where it violates the instruction.
- Edit: The Editor proposes a minimal replacement span—on average, fewer than two tokens—to put the model back on track.
- Resume: The model continues generating from that corrected prefix. This can repeat until the trace is compliant.
- Train: The model is trained using a cross-entropy loss only on those edited tokens.
The authors tested two versions:
- Offline: Training on a fixed dataset of edited traces.
- Online: A "DAgger-style" approach where the model's current failures are continuously edited and used for training, allowing the supervision to evolve as the model improves.
Datasets
- ReasonIF: The primary benchmark, featuring tasks like "Uppercase only," "JSON formatting," "Multilinguality," and "Word limits" applied to math and logic problems (AIME, GSM8K, etc.).
- IFEval: Used to see if teaching a "draft-review-revise" reasoning procedure improves final answer accuracy on constrained prompts.
- IMO-AnswerBench: A set of high-level math olympiad problems used to check if the training process accidentally "broke" the model's core mathematical abilities.
Key Insights and Ideas
- Surgical vs. Blunt Force: By only training on the edits, the model retains its "native" reasoning patterns. It learns to avoid the specific mistake without being forced to adopt a teacher's entire linguistic style.
- The Power of the Prefix: A tiny correction early in a trace can completely redirect the model's downstream logic. This confirms that "first-error" supervision is highly efficient.
- Credit Assignment Solved: Unlike RL, which guesses which part of a long chain was bad, this method points exactly to the "critical token" that caused the failure.
- Online vs. Offline Trade-offs: The Online version is the king of in-domain performance (reaching 97% accuracy), but the Offline version generalizes better to instructions it has never seen before (68.4% vs 34%).
Key Results
The researchers evaluated their method using Qwen3-32B and Nemotron-3-Nano.
- Instruction Compliance: On the ReasonIF benchmark, the Online edit-based model achieved 97% accuracy, vastly outperforming standard SFT (40.3%) and RL-based GRPO (64%).
- Capability Preservation: One of the most striking results was that while SFT-GPT (distilling from a stronger model) caused mathematical performance on IMO-AnswerBench to tank (dropping from 61% to 43%), the Online edit-based model stayed at 59%, nearly matching the base model.
- Generalization: The Offline model showed a remarkable ability to follow "unseen" instructions (like being asked to organize reasoning into specific sections), outperforming even the GPT-distilled baseline.
Limitations and Future Work
- The "Oracle" Problem: The method currently relies on a very strong external model (GPT-5) or complex scripts to act as the Editor. A major future direction is self-improvement, where a model learns to edit its own thoughts.
- Proxy Tasks: The ReasonIF categories (like "no commas") are useful proxies for control, but they aren't "natural" reasoning. The authors want to apply this to fixing substantive logical errors in the future.
- Detection Bottleneck: In the IFEval experiments, the model often failed not because it couldn't fix an error, but because it couldn't detect that it had made one in its own draft. Improving self-monitoring is a key next step.
Summary updated