Synthetic Error Injection Fails to Elicit Self-Correction In Language Models

David X. Wu, Shreyas Kapur, Anant Sahai, Stuart Russell

TL;DR

Can we teach language models to self-correct their reasoning flaws using cheap supervised fine-tuning instead of expensive reinforcement learning? This paper from UC Berkeley demonstrates that a popular robotics-inspired approach—injecting synthetic errors into training data—fundamentally fails because models cannot generalize error-correction skills from artificial mistakes to their own natural, context-dependent errors.

What Problem Does This Paper Address?

State-of-the-art reasoning models (like OpenAI's o1 or DeepSeek-R1) generate long chains of thought (CoT) that allow them to catch and fix their own mistakes before outputting a final answer. Currently, this emergent self-correction capability is unlocked via Reinforcement Learning (RL), which is notoriously complex and computationally expensive.

In robotics and autonomous driving, researchers often bypass RL using a clever supervised trick: they artificially perturb data during training (e.g., making a self-driving car drift off-center) and supervise the model to steer back into the lane. The authors of this paper wanted to know if a similar approach could work for LLMs. Can we simply inject synthetic errors into a model's reasoning chains during supervised fine-tuning (SFT) to teach it how to recover from mistakes?

Methods

The authors introduce Error Injection Fine-Tuning (EIFT). Instead of training models exclusively on flawless "golden" reasoning paths, they mix in data where 20% of the chains contain artificial mistakes.

When an error is injected, a correct reasoning step is replaced with a three-part sequence:

  1. The Erroneous Step: An intentionally flawed calculation or move.
  2. The Error Recognition Step: An explicit trigger phrase: "Ah! I made a mistake."
  3. The Error Correction Step: The original, correct reasoning step.

During training, the loss for the injected error step itself is masked out, forcing the model to focus entirely on learning how to recognize and correct the mistake.

Figure 1. An illustration of the synthetic error injection process. The (Left) panel shows a correct Chain-of-Thought (CoT) trace used for standard fine-tuning (FT). The (Right) panel details the Error Injection Fine-Tuning (EIFT) methodology, where a correct step from a golden CoT is replaced by a three-part sequence: (1) a synthetically injected erroneous step (e.g., a “Carry Error”), (2) an explicit error recognition step (“Ah! I made a mistake.”), and (3) the original correct step as a supervised correction. As indicated by the “Include in Loss?” column, the loss is computed for the recognition and correction steps but masked for the synthetically injected error.

Datasets

To maintain precise control over error distributions, the authors evaluated EIFT on two algorithmic tasks across three open-source base models (Qwen2.5-Math-1.5B-Instruct, Gemma-3-1B-it, and Llama-3.2-1B-Instruct):

  1. Multiplication: Random 4-digit multiplication problems, utilizing six distinct synthetic error types (such as carry errors or single-digit swaps).
  2. Sudoku: 4×4 Sudoku puzzles restricted to "naked single" moves. This task was specifically chosen to introduce a strong solver-verifier gap—meaning it is computationally trivial for a model to verify if a number violates a row/column constraint, even if generating the correct move is difficult.

Key Insights and Ideas

  • The Distribution Mismatch Trap: The core reason EIFT fails is a subtle distribution shift. Even though the synthetic error generator had near-perfect coverage of the types of errors the models naturally made (>95% coverage), the models only learned to correct the exact, synthetic variations encountered during training. They failed to generalize this correction capability to their own "on-policy" errors (the context-dependent mistakes they naturally make).
  • The "Parroting" Phenomenon: One of the most surprising findings was that when models encountered their own natural errors, they would frequently output the recognition token ("Ah! I made a mistake.") but then immediately repeat the exact same incorrect step they just claimed to catch. This implies that the model's internal verification circuit can identify that something is wrong, but it isn't properly integrated with the generation circuit to compute an alternative path.
  • The Solver-Verifier Gap Helps (But Not Enough): EIFT provided a larger performance boost on Sudoku than on multiplication. Because verifying a Sudoku constraint is much easier than calculating a 4-digit multiplication step from scratch, the model had an easier time leveraging the error-injection data, though it still fell short of RL-like performance.

Key Results

  • Negligible Accuracy Gains: EIFT provided almost zero final accuracy improvement for multiplication across all models. For Sudoku, it provided a modest ~10% boost for Qwen and Llama, but absolutely no benefit for Gemma.
  • Precipitous Performance Drops: While the fine-tuned models became excellent at correcting synthetic errors, their performance plummeted when facing their own natural (on-policy) errors. For instance, Qwen2.5’s error recognition rate on Sudoku dropped from 94% on synthetic errors to a dismal 8% on on-policy errors.
  • On the multiplication task, Qwen2.5 retained a decent recognition rate (78%) on its own errors, but its actual correction rate plummeted from 99% on synthetic errors to just 40% on on-policy errors.

Limitations and Future Work

A primary limitation of this study is its focus on relatively simple, toy algorithmic tasks (multiplication and 4×4 Sudoku). It remains an open question whether these distribution shifts behave differently in complex natural language reasoning, creative writing, or code generation.

The authors conclude that for supervised self-correction to work, the training data must precisely match the model's innate, context-dependent error modes. However, generating a dataset of a model's own natural errors requires an automated, highly accurate verifier (like a process reward model), which brings the complexity and compute cost right back to the realm of RL—defeating the original appeal of a cheap SFT alternative.

Summary updated

Abstract

Reinforcement learning has become the dominant paradigm for eliciting reasoning and self-correction capabilities in large language models, but its computational expense motivates exploration of alternatives. Inspired by techniques from autonomous driving and robotics, we investigate whether supervised learning with synthetic error injection can induce self-correction abilities in language models. Our approach inserts artificial errors into reasoning chains, masks them, and supervises the model to recognize and correct these mistakes. Despite the intuitive appeal of this method, we find that it fails to significantly improve performance even on simple synthetic tasks across multiple models. Moreover, even when the model catches its own error, it often parrots the original mistake. We find that the distribution shift of synthetic errors to on-policy errors significantly degrades the error-correction capabilities of the fine-tuned model, even with good synthetic coverage of on-policy errors. Our results help explain why on-policy reinforcement learning methods have proven uniquely effective for eliciting self-correction.
BibTeX
@misc{wu2025synthetic,
  author = {David X. Wu and Shreyas Kapur and Anant Sahai and Stuart Russell},
  title = {{Synthetic Error Injection Fails to Elicit Self-Correction In Language Models}},
  year = {2025},
  eprint = {2512.02389},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.02389},
}
Figure