Reasoning Models Sometimes Output Illegible Chains of Thought

Arun Jose

TL;DR

This paper investigates why large language models (LLMs) trained with outcome-based reinforcement learning (RL) often produce "illegible" chains of thought (CoTs) – reasoning steps that are difficult for humans and AI to understand. The study finds that this illegibility is a common byproduct of RL training, even though the final answers remain perfectly legible. This phenomenon poses a significant challenge for AI safety and monitoring efforts, as it obscures the model's decision-making process.

What Problem Does This Paper Address?

As LLMs become more capable of complex reasoning, particularly through techniques like Chain-of-Thought (CoT) prompting, there's a growing interest in understanding how they arrive at their answers. This is crucial for AI safety, as monitoring a model's reasoning process (its CoT) could help detect malicious behavior or misalignment. However, the effectiveness of this monitoring hinges on the CoT being both "legible" (understandable) and "faithful" (accurately representing the model's reasoning).

This paper tackles the problem that outcome-based RL, a common training method for advanced reasoning LLMs, appears to be reducing the legibility of these CoTs. The authors are concerned that if models develop opaque reasoning processes, it could undermine our ability to monitor and control them, especially as models become more powerful and their reasoning more complex. They aim to understand if this illegibility is a bug or a feature, and what its implications are for AI safety.

Methods

The researchers evaluated the legibility of CoTs from 14 different LLMs, including models trained with outcome-based RL (like DeepSeek R1) and their non-reasoning counterparts. To quantify legibility, they used GPT-4o as an automated grader, scoring CoTs on a scale of 1-9 (higher scores indicating less legible).

They employed several experimental approaches:

  • Legibility Scoring: GPT-4o was prompted to score CoT segments for legibility.
  • Prefill Experiments: To test the usefulness of illegible reasoning, they truncated CoTs at the point they became illegible and used this as input for another instance of the model, forcing it to produce a final answer. This allowed them to measure performance degradation when illegible reasoning was removed.
  • Resampling Experiments: They explored the correlation between legibility and performance by resampling responses to questions.
  • Question Hardness Analysis: They categorized questions by difficulty and analyzed how legibility varied with task complexity.
  • Temperature Effects: They investigated how sampling temperature during generation influenced CoT legibility.

Datasets

The primary dataset used for evaluation was the GPQA-Diamond dataset. This is a challenging dataset of 198 multiple-choice questions in biology, physics, and chemistry. The authors modified the dataset by removing answer choices to increase the difficulty and create more complex reasoning tasks.

Key Insights and Ideas

  • RL's Illegibility Problem: Outcome-based RL training, while improving performance, often leads to CoTs that are difficult for both humans and AI monitors to understand. This is a widespread phenomenon, observed across many reasoning models, with the notable exception of Claude models.
  • Legible Answers, Illegible Reasoning: Models often "snap back" to perfect legibility for their final answers, even when the preceding reasoning steps are opaque. This suggests a disconnect between the internal reasoning process and the final output.
  • Illegible Reasoning is Useful (But Not Necessarily Better): Prefill experiments showed that removing illegible portions of the CoT significantly degraded model performance (accuracy dropped by 53% when forced to use only legible parts). This indicates that the illegible reasoning is not just noise but contributes to the model's ability to arrive at the correct answer. However, resampling experiments found no consistent correlation between legibility and performance, suggesting that illegible reasoning isn't inherently better than legible reasoning, but rather a useful component of the model's process.
  • Harder Questions, More Illegibility: Legibility decreases as question difficulty increases. This implies that as models tackle more complex problems, their reasoning becomes more opaque, which is precisely when monitoring is most critical.
  • Model Size and Illegibility: Larger models within a training paradigm tend to produce more illegible reasoning, suggesting it's not simply a matter of limited model capacity but rather an emergent property of scaling RL for reasoning.
  • Potential Hypotheses: The paper proposes several explanations for illegible CoTs, including:
  • Complex Steganography: Efficient reasoning pathways that compress information for performance.
  • Meaningless RL Artifact: Illegible text is entirely unrelated to the reasoning process (partially refuted by performance drops).
  • Vestigial Reasoning: RL reinforces any CoT leading to correct answers, even if parts are not causally essential for the final step but are useful for sequential processing. This is the most favored hypothesis, suggesting illegible tokens are "vestigially useful" for guiding the model's internal state.

Key Results

  • Widespread Illegibility: 13 out of 14 models evaluated showed meaningful illegibility in their CoTs, with R1, R1-Zero, and QwQ exhibiting the highest scores.
  • Performance Drop: Forcing a model (QwQ) to use only legible portions of its CoT resulted in a significant performance drop, with correct answers decreasing from 24.6% to 11.5% (a 53% decrease).
  • Question Difficulty Impact: Legibility scores decreased with question hardness. For example, R1 showed a 23% increase in illegibility between easy and hard questions.
  • Temperature Sensitivity: For R1, sampling at temperature 0 significantly reduced CoT illegibility, suggesting that its legibility training was brittle and primarily affected top logprobs rather than fundamentally altering its reasoning.

Limitations and Future Work

The authors acknowledge several limitations:

  • Causal Mechanisms: They cannot definitively determine the exact causal mechanisms behind illegible CoTs from their observations alone. More controlled experiments are needed.
  • Claude's Legibility: The consistent legibility of Claude models across all tested conditions, even with outcome-based RL, suggests that illegibility is not an inevitable consequence of RL but rather a likely one. This contrast highlights the need for further investigation into what makes Claude different.
  • Dataset Specificity: The evaluation relied on GPQA-Diamond, which, while challenging, might not fully represent the range of tasks LLMs encounter.

Future work suggested includes:

  • Directly comparing models with varying degrees of legibility on a broader range of tasks.
  • Investigating specific post-training techniques that maintain legibility without sacrificing performance.
  • Using white-box methods to gain a deeper understanding of what illegible CoTs are doing.
  • Exploring how to develop more robust monitoring systems that can handle illegible reasoning, especially for harder questions where oversight is most valuable.

Summary updated

BibTeX
@inproceedings{jose2025reasoning,
  author = {Arun Jose},
  title = {{Reasoning Models Sometimes Output Illegible Chains of Thought}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=w1TjXJk846},
}
Figure