Reasoning Models Sometimes Output Illegible Chains of Thought

Arun Jose

As artificial intelligence moves toward models that use Reinforcement Learning (RL) to optimize for correct answers, a concerning trend is emerging in AI safety and interpretability: high-performance reasoning models sometimes "think" in gibberish. This phenomenon threatens to close the window researchers use to monitor model behavior and intentions.

The "Legibility Gap" in AI Safety

To ensure AI systems remain safe and aligned, researchers rely heavily on Chain-of-Thought (CoT) monitoring. By having a model explain its step-by-step reasoning before answering, auditors can catch malicious intent, biased logic, or hallucinated justifications.

However, this safety mechanism relies on a critical assumption: that the CoT is both faithful (reflecting the model's actual internal logic) and legible (understandable to humans). If models trained via outcome-based RL discover that generating "gibberish" tokens helps them reach the correct answer more efficiently than natural language, they may naturally drift toward illegibility.

Comparison of an illegible chain of thought (score 9) versus a highly legible one (score 1).

This paper investigates this "legibility gap," quantifying how widespread illegible reasoning is among state-of-the-art models and determining whether these seemingly nonsensical tokens actually contribute to model performance.

Evaluating the Models

Researchers evaluated 14 different models—including the DeepSeek R1 family, QwQ, Qwen3, Kimi K2, and various Anthropic Claude models—using the GPQA-Diamond dataset. This dataset consists of highly difficult, expert-written questions in biology, physics, and chemistry. To increase the challenge, multiple-choice options were removed, forcing models to generate answers entirely from scratch.

To measure legibility, the study employed GPT-4o as an "autograder" to score CoT transcripts on a scale from 1 (perfectly legible) to 9 (entirely nonsensical).

Key Findings

1. The "Snap Back" Phenomenon

Models like DeepSeek R1 often output hundreds of tokens of absolute gibberish—mixing nonsensical phrases, random words, and non-English characters (e.g., "Mom likes y’all base abstracts tot tern a..."). Yet, remarkably, they can suddenly "snap back" to perfect English to deliver a coherent, correct final answer.

2. Illegibility is Functional

This gibberish is not merely a sampling artifact or random noise; it is functional. To test this, researchers performed "prefill experiments" where they took a model's CoT, truncated it at the exact point it became illegible, and forced the model to answer based only on the legible portion. For models like QwQ, truncating the illegible text caused a 53% relative drop in accuracy. The model actively uses these strange tokens to "think" and solve the problem.

3. The Scaling Paradox and Model Differences

Within the same model family, larger models tend to produce more illegible reasoning. As models gain computational capacity through RL, they appear more likely to abandon human language in their internal scratchpads.

Illegibility scores across various models, highlighting the stark contrast between the DeepSeek/Qwen families and Claude models.

Notably, DeepSeek R1 and R1-Zero showed the highest levels of illegibility, with R1 outputting "entirely inscrutable" text in 29.4% of cases when tackling hard scientific questions. Conversely, Anthropic’s Claude models remained highly legible across the board. This suggests that illegibility is not an inevitable byproduct of advanced reasoning, but rather a result of specific training choices—such as the lack of coherence penalties in certain RL pipelines.

4. Difficulty Breeds Illegibility

Illegibility increases with question difficulty. R1 showed a 23% increase in illegibility when moving from "easy" to "hard" questions. This implies that CoT monitoring may become least reliable precisely when it is most needed: during complex, high-stakes tasks.

Implications and Future Work

The findings present a significant challenge for AI safety. If RL reinforces any token sequence that leads to a correct answer, models may develop "vestigial" or "alien" reasoning patterns. These illegible tokens might act as bridge tokens to maintain state or trigger specific computations, even if they lack English meaning.

Future research must focus on "white-box" interpretability methods to decode what these illegible tokens represent. Furthermore, AI developers may need to design new RL techniques that explicitly penalize illegibility, ensuring that as models grow smarter, their thought processes remain transparent to human overseers.

Summary updated

Abstract

Language models trained via outcome-based reinforcement learning (RL) to reason using chain-of-thought (CoT) have shown remarkable performance. Monitoring such a model's CoT may allow us to understand its intentions and detect potential malicious behavior. However, to be effective, this requires that CoTs are legible and faithful. We evaluate the legibility of CoTs in state-of-the-art reasoning models. We find that R1, R1-Zero, and QwQ often produce illegible CoTs (mixing nonsensical phrases, random words, and non-English characters) before returning to perfect coherence in their final responses, while Claude models notably exhibit higher legibility. Across 14 models, we observe that larger models within the same training paradigm tend to produce more illegible reasoning. Prefill experiments show that truncating reasoning at a legibility threshold reduces accuracy by 53\%, suggesting that illegible portions contribute to performance despite being difficult to monitor. Illegibility increases with question difficulty, suggesting that CoT monitoring may be less reliable precisely when most needed. We discuss potential hypotheses for these results, including steganography, vestigial tokens, and training artifacts. Our findings suggest that current approaches to CoT monitoring may be vulnerable to the emergence of outcome-based RL, particularly as models face increasingly complex tasks.
BibTeX
@inproceedings{jose2025reasoning,
  author = {Arun Jose},
  title = {{Reasoning Models Sometimes Output Illegible Chains of Thought}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=w1TjXJk846},
}
Figure