Training LLMs to Follow Reasoning Instructions with Minimal Edits

Alexandra Yakovleva, Luat Gia Khoi Nguyen, Juho Kannala, Harri Valpola, Alexander IlinarXiv 2026 · arXiv

TL;DR

Instead of training LLMs to imitate entire reasoning "trajectories" or rewarding them for final answers, this paper proposes edit-based training. By identifying the exact point where a model first violates a reasoning instruction and providing a minimal textual correction (often just a token or two), researchers can steer model behavior with surgical precision while preserving its original problem-solving capabilities.

What Problem Does This Paper Address?

As we move toward "System 2" models that think before they speak, we need more control over how they reason. A user might want a model to "reason in Chinese but answer in English" or "check every assumption before proceeding."

Currently, we have two main ways to train this:

  1. Supervised Fine-Tuning (SFT): We give the model a full "gold standard" reasoning trace to copy. The problem? The model often imitates incidental quirks of the teacher model rather than just the logic, leading to "entropy collapse" where the model loses its own diverse ways of solving problems.
  2. Reinforcement Learning (RL): We give the model a reward based on the final outcome. The problem? This has poor credit assignment. If a model writes a 500-word reasoning trace and fails because of one tiny mistake in the middle, RL penalizes the whole thing, even the parts that were actually good.

The authors seek a middle ground: textual supervision that is sparse, localized, and focuses only on the specific failure point.

Methods

The core of the paper is the edit-based training procedure. It works like a surgical intervention:

  1. Sample: The model generates a reasoning trace.
  2. Identify: An "Editor" (which can be a script or a stronger LLM like GPT-5) scans the trace to find the first point where it violates the instruction.
  3. Edit: The Editor proposes a minimal replacement span—on average, fewer than two tokens—to put the model back on track.
  4. Resume: The model continues generating from that corrected prefix. This can repeat until the trace is compliant.
  5. Train: The model is trained using a cross-entropy loss only on those edited tokens.
The iterative editing process: red circles mark violations, green circles mark the minimal edits that steer the model back to compliance.

The authors tested two versions:

  • Offline: Training on a fixed dataset of edited traces.
  • Online: A "DAgger-style" approach where the model's current failures are continuously edited and used for training, allowing the supervision to evolve as the model improves.

Datasets

  • ReasonIF: The primary benchmark, featuring tasks like "Uppercase only," "JSON formatting," "Multilinguality," and "Word limits" applied to math and logic problems (AIME, GSM8K, etc.).
  • IFEval: Used to see if teaching a "draft-review-revise" reasoning procedure improves final answer accuracy on constrained prompts.
  • IMO-AnswerBench: A set of high-level math olympiad problems used to check if the training process accidentally "broke" the model's core mathematical abilities.

Key Insights and Ideas

  • Surgical vs. Blunt Force: By only training on the edits, the model retains its "native" reasoning patterns. It learns to avoid the specific mistake without being forced to adopt a teacher's entire linguistic style.
  • The Power of the Prefix: A tiny correction early in a trace can completely redirect the model's downstream logic. This confirms that "first-error" supervision is highly efficient.
  • Credit Assignment Solved: Unlike RL, which guesses which part of a long chain was bad, this method points exactly to the "critical token" that caused the failure.
  • Online vs. Offline Trade-offs: The Online version is the king of in-domain performance (reaching 97% accuracy), but the Offline version generalizes better to instructions it has never seen before (68.4% vs 34%).

Key Results

The researchers evaluated their method using Qwen3-32B and Nemotron-3-Nano.

  • Instruction Compliance: On the ReasonIF benchmark, the Online edit-based model achieved 97% accuracy, vastly outperforming standard SFT (40.3%) and RL-based GRPO (64%).
  • Capability Preservation: One of the most striking results was that while SFT-GPT (distilling from a stronger model) caused mathematical performance on IMO-AnswerBench to tank (dropping from 61% to 43%), the Online edit-based model stayed at 59%, nearly matching the base model.
  • Generalization: The Offline model showed a remarkable ability to follow "unseen" instructions (like being asked to organize reasoning into specific sections), outperforming even the GPT-distilled baseline.
Table 1: Overview of the main results (higher is better). The first three rows report overall reasoning instruction-following accuracy (%) in three settings: in-domain, on out-of-domain prompts, and under unseen reasoning instructions. The last two rows measure preservation of capabilities not targeted by training: final-answer accuracy (%) on the ReasonIF tasks and on IMO-AnswerBench, both evaluated with the reasoning instruction removed. Trained systems report mean ± standard deviation across three training runs; among them, the best result in each row is shown in bold, together with any result whose standard deviation covers it. Per-category breakdowns of the instruction-following results are given in Tables 3, 5, and 6.

Limitations and Future Work

  • The "Oracle" Problem: The method currently relies on a very strong external model (GPT-5) or complex scripts to act as the Editor. A major future direction is self-improvement, where a model learns to edit its own thoughts.
  • Proxy Tasks: The ReasonIF categories (like "no commas") are useful proxies for control, but they aren't "natural" reasoning. The authors want to apply this to fixing substantive logical errors in the future.
  • Detection Bottleneck: In the IFEval experiments, the model often failed not because it couldn't fix an error, but because it couldn't detect that it had made one in its own draft. Improving self-monitoring is a key next step.
Table 2 highlights that while the procedure helps, the model still struggles to detect its own violations in complex multi-constraint scenarios.

Summary updated

Abstract

Large language models increasingly rely on extended reasoning traces, yet the training signals used to shape this behavior remain coarse. Supervised fine-tuning on full trajectories forces the model to imitate an entire demonstration, overconstraining its output distribution, while trajectory-level reinforcement learning provides only weak credit assignment, rewarding or penalizing whole completions without identifying where they succeed or fail. We propose edit-based training, a general framework for learning from a model's own trajectories via localized textual corrections. Given a sampled completion, an editor identifies the first point of failure under a specified criterion and proposes a minimal replacement span. The model is then supervised only on these edited spans. By design, this concentrates supervision where the model's reasoning first breaks down and aims to preserve its existing reasoning patterns elsewhere. The method applies both offline and online, with the latter continually adapting supervision to the model's evolving failure modes. We instantiate this approach on reasoning instruction following with Qwen3-32B on the ReasonIF benchmark. Compared to full-trajectory imitation and trajectory-level reinforcement learning baselines, edit-based training substantially improves compliance with the reasoning instruction while largely preserving the model's skills on untargeted tasks. These results suggest that localized correction is an effective complement to these approaches for shaping how LLMs reason.
BibTeX
@misc{yakovleva2026training,
  author = {Alexandra Yakovleva and Luat Gia Khoi Nguyen and Juho Kannala and Harri Valpola and Alexander Ilin},
  title = {{Training LLMs to Follow Reasoning Instructions with Minimal Edits}},
  year = {2026},
  eprint = {2611.26342},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26342},
}
Figure