Humans Perceive Wrong Narratives from AI Reasoning Texts

Mosh Levy, Zohar Elyoseph, Yoav GoldbergarXiv 2025 · DOI · arXiv

Introduction: The Narrative-Process Gap

As Large Language Models (LLMs) evolve into "Large Reasoning Models" (such as DeepSeek-R1 or OpenAI’s o1), they increasingly produce long, step-by-step chains of thought before arriving at a final answer. For researchers and users alike, these reasoning texts are often viewed as a "window" into the model's logic—a way to debug, verify, or trust the AI's output.

However, this research identifies a fundamental disconnect: the narrative (the story a human constructs while reading the text) is often entirely decoupled from the process (how the model actually uses those tokens to compute the next step). If humans cannot identify which previous steps actually "cause" a later one, then reasoning text fails as a reliable explanation of AI behavior.

Conceptual framework illustrating the disconnect between the AI's internal computational process and the narrative perceived by humans.

Methodology: The AI Narrative Test

To quantify this gap, the researchers developed a rigorous framework to compare human intuition against ground-truth AI causality:

  1. Causal Step Intervention: To establish "ground truth," the authors systematically removed individual steps from a reasoning chain and observed if subsequent steps changed. A causal link was defined as a change in Step B following the removal of Step A.
  2. Human Evaluation: These links were converted into a multiple-choice test. Participants were shown a math problem and a reasoning chain, then asked to identify which of four candidate steps was the causal driver for a specific "target step."
  3. Datasets and Models: The study utilized 15 problems from the GSM8K (Grade School Math) dataset. Reasoning chains were generated using state-of-the-art models: DeepSeek-R1 (671B) and Qwen-3 (32B).
  4. Participants: 80 educated adults, many with STEM backgrounds, completed 50 questions each via the Prolific platform.

Results: A Universal Failure in Intuition

The results were startling: human performance was remarkably poor across the board. Participants achieved an average accuracy of only 29%. Given that the test used a four-option multiple-choice format, the baseline for blind guessing was 25%.

Distribution of participant accuracy across the study, showing a mean performance of 29.0%, only slightly above the 25% random chance baseline.

Crucially, this failure was universal. Not a single participant exceeded 50% accuracy. Furthermore, demographic factors—such as having a PhD, a STEM background, or extensive experience with AI—had no significant impact on performance. Everyone struggled to map the model's actual dependencies.

Discussion: The Anthropomorphic Trap

The study suggests that humans fall into an "anthropomorphic trap." We read AI reasoning through the lens of human discourse, assuming that if a model says "Wait, let me recalculate," that specific moment is a logical pivot. In reality, the model might be ignoring that "aha moment" entirely while relying on a seemingly trivial sentence several steps back.

This leads to "Shared Illusions." On half of the questions, the majority of participants agreed on the same incorrect answer. This indicates that humans share a consistent, but fundamentally wrong, narrative about how AI "thinks." Collective agreement in these cases does not signal correct understanding, but rather a shared misunderstanding of the computational artifact.

Conclusion and Future Directions

The authors argue that we must stop treating reasoning text as a transparent window into AI logic. Instead, it should be viewed as a "computational artifact"—something the model produces that may help it reach a correct answer, but which does not explain how it got there in a way humans can intuitively grasp.

This research calls for a "new science" of AI interpretation. Moving forward, we need better tools to visualize and verify the actual data flow within reasoning chains rather than relying on human-readable narratives that may be misleading.

Summary updated

Abstract

A new generation of AI models generates step-by-step reasoning text before producing an answer. This text appears to offer a human-readable window into their computation process, and is increasingly relied upon for transparency and interpretability. However, it is unclear whether human understanding of this text matches the model's actual computational process. In this paper, we investigate a necessary condition for correspondence: the ability of humans to identify which steps in a reasoning text causally influence later steps. We evaluated humans on this ability by composing questions based on counterfactual measurements and found a significant discrepancy: participant accuracy was only 29%, barely above chance (25%), and remained low (42%) even when evaluating the majority vote on questions with high agreement. Our results reveal a fundamental gap between how humans interpret reasoning texts and how models use it, challenging its utility as a simple interpretability tool. We argue that reasoning texts should be treated as an artifact to be investigated, not taken at face value, and that understanding the non-human ways these models use language is a critical research direction.
BibTeX
@misc{levy2025humans,
  author = {Mosh Levy and Zohar Elyoseph and Yoav Goldberg},
  title = {{Humans Perceive Wrong Narratives from AI Reasoning Texts}},
  year = {2025},
  eprint = {2508.16599},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.16599},
}
Figure