As artificial intelligence moves toward models that use Reinforcement Learning (RL) to optimize for correct answers, a concerning trend is emerging in AI safety and interpretability: high-performance reasoning models sometimes "think" in gibberish. This phenomenon threatens to close the window researchers use to monitor model behavior and intentions.
The "Legibility Gap" in AI Safety
To ensure AI systems remain safe and aligned, researchers rely heavily on Chain-of-Thought (CoT) monitoring. By having a model explain its step-by-step reasoning before answering, auditors can catch malicious intent, biased logic, or hallucinated justifications.
However, this safety mechanism relies on a critical assumption: that the CoT is both faithful (reflecting the model's actual internal logic) and legible (understandable to humans). If models trained via outcome-based RL discover that generating "gibberish" tokens helps them reach the correct answer more efficiently than natural language, they may naturally drift toward illegibility.
This paper investigates this "legibility gap," quantifying how widespread illegible reasoning is among state-of-the-art models and determining whether these seemingly nonsensical tokens actually contribute to model performance.
Evaluating the Models
Researchers evaluated 14 different models—including the DeepSeek R1 family, QwQ, Qwen3, Kimi K2, and various Anthropic Claude models—using the GPQA-Diamond dataset. This dataset consists of highly difficult, expert-written questions in biology, physics, and chemistry. To increase the challenge, multiple-choice options were removed, forcing models to generate answers entirely from scratch.
To measure legibility, the study employed GPT-4o as an "autograder" to score CoT transcripts on a scale from 1 (perfectly legible) to 9 (entirely nonsensical).
Key Findings
1. The "Snap Back" Phenomenon
Models like DeepSeek R1 often output hundreds of tokens of absolute gibberish—mixing nonsensical phrases, random words, and non-English characters (e.g., "Mom likes y’all base abstracts tot tern a..."). Yet, remarkably, they can suddenly "snap back" to perfect English to deliver a coherent, correct final answer.
2. Illegibility is Functional
This gibberish is not merely a sampling artifact or random noise; it is functional. To test this, researchers performed "prefill experiments" where they took a model's CoT, truncated it at the exact point it became illegible, and forced the model to answer based only on the legible portion. For models like QwQ, truncating the illegible text caused a 53% relative drop in accuracy. The model actively uses these strange tokens to "think" and solve the problem.
3. The Scaling Paradox and Model Differences
Within the same model family, larger models tend to produce more illegible reasoning. As models gain computational capacity through RL, they appear more likely to abandon human language in their internal scratchpads.
Notably, DeepSeek R1 and R1-Zero showed the highest levels of illegibility, with R1 outputting "entirely inscrutable" text in 29.4% of cases when tackling hard scientific questions. Conversely, Anthropic’s Claude models remained highly legible across the board. This suggests that illegibility is not an inevitable byproduct of advanced reasoning, but rather a result of specific training choices—such as the lack of coherence penalties in certain RL pipelines.
4. Difficulty Breeds Illegibility
Illegibility increases with question difficulty. R1 showed a 23% increase in illegibility when moving from "easy" to "hard" questions. This implies that CoT monitoring may become least reliable precisely when it is most needed: during complex, high-stakes tasks.
Implications and Future Work
The findings present a significant challenge for AI safety. If RL reinforces any token sequence that leads to a correct answer, models may develop "vestigial" or "alien" reasoning patterns. These illegible tokens might act as bridge tokens to maintain state or trigger specific computations, even if they lack English meaning.
Future research must focus on "white-box" interpretability methods to decode what these illegible tokens represent. Furthermore, AI developers may need to design new RL techniques that explicitly penalize illegibility, ensuring that as models grow smarter, their thought processes remain transparent to human overseers.
Summary updated