When Reasoning looks like Hidden Understanding: How Chain-of-Thought Fosters Anthropomorphism, Perceptions of Epistemic Superiority, and Cognitive Deference

Oliver Lack, Lana Tikhomirov, Humphrey Thompson, Carolyn Semmler, Melissa McCraddenarXiv 2026 · arXiv

This position paper from researchers at the Australian Institute for Machine Learning and the University of Adelaide challenges the current trend of using "Chain-of-Thought" (CoT) as a tool for AI explainability. The authors argue that CoT acts as a psychological "vehicle" for anthropomorphism, leading users to grant AI unearned epistemic authority and defer to its judgment even when it is wrong.

TL;DR

Chain-of-Thought (CoT) outputs are often mistaken for genuine machine "reasoning," but they are frequently unfaithful to the model's actual internal processes. This paper warns that user-facing CoT fosters a dangerous "cognitive deference," where humans stop critically evaluating AI outputs, particularly in high-stakes fields like medicine.

What Problem Does This Paper Address?

As generative AI becomes ubiquitous, a belief system is emerging that equates linguistic fluency with "understanding." A key feature driving this is Chain-of-Thought (CoT)—the intermediate steps a model generates before providing a final answer.

The authors identify a critical gap: while CoT is technically useful for improving model accuracy, it is being marketed and perceived as a transparent "explanation" of the model's logic. The problem is that these "thoughts" are often post-hoc justifications rather than faithful records of the model's causal process. This creates a "fallacy of composition," where users see a model perform well on a narrow task and incorrectly assume it possesses broad, human-like cognitive competence.

Methods

This is a theoretical and conceptual analysis rather than an empirical study. The authors synthesize research from cognitive science, philosophy, and AI ethics to build a framework for how humans interact with "reasoning" models.

Key conceptual pillars include:

  • Epistemic Authority: The degree to which an entity is treated as a source of knowledge.
  • The Pre-emption Thesis: The psychological phenomenon where a trusted authority's reasoning replaces one's own independent weighing of evidence.
  • Cognitive Deference: The shifting of complex judgment tasks from the human to the AI.
  • Case Study Analysis: The paper applies these concepts to the medical field to illustrate the risks of AI-aided clinical decision-making.

Datasets

Not discussed in the paper. As a position paper, it does not introduce new datasets or benchmarks, though it cites numerous empirical studies (e.g., systematic reviews of AI vs. clinicians) to support its arguments.

Key Insights and Ideas

  • Anthropomorphism as a Vehicle: The authors argue that anthropomorphism isn't the driver of trust, but the vehicle. By making the AI seem "cognitively familiar" (using words like "I think" or "planning"), it makes the idea of deferring to the machine seem plausible to the user.
  • The "Machine Heuristic" Amalgamation: Historically, we viewed machines as objective and rule-bound (the "machine heuristic"). Now, we are combining that with the perception of human-like reasoning. This makes generative AI appear uniquely—and dangerously—authoritative.
  • Unfaithful Explanations: CoT is often "unfaithful," meaning the steps shown to the user do not actually represent how the model reached its conclusion. In this light, CoT is better described as persuasion rather than explanation.
  • The Fallacy of Composition: Users often commit this fallacy by inferring that because a model can "reason" through a summary, it has the "diagnostic competence" to handle a complex medical case.

Key Results

While the paper is qualitative, it highlights several concerning findings from cited literature:

  • The Trust Paradox: The presence of an explanation (even a random or incorrect one) significantly increases the likelihood that a user will follow a wrong AI recommendation.
  • Clinical Over-reliance: In medical diagnostic experiments, laypeople showed high "automation bias" when presented with LLM explanations, whereas only highly experienced physicians remained resilient to incorrect AI advice.
  • Verification Failure: The authors cite data showing that in Google’s "AI Overviews," less than 1% of users click the source links to verify the AI's summary, indicating high levels of cognitive deference.

Limitations and Future Work

The authors acknowledge that CoT remains a valid and valuable research direction for improving model performance and internal moderation. However, they note two major hurdles for future work:

  1. Faithfulness: Even if CoT becomes perfectly faithful to the model's logic, it doesn't guarantee that users will reject incorrect outputs.
  2. Holistic Judgment: In fields like medicine, clinical judgment requires integrating diverse data points (patient values, physical exams) that a text-based CoT cannot capture.

Recommendation: The authors conclude that the practice of asking models to provide user-facing "reasons" should be actively discouraged. Instead, AI literacy training should explicitly teach users to view CoT as a potential source of persuasion rather than a window into the machine's "mind."

Summary updated

Abstract

As AI-based products permeate society, specifically systems using generative models, a belief system is emerging about what these products are and what they can and cannot do. One common belief concerns the perception of the form of language as evidence of 'understanding' language. As these generative models reproduce linguistic patterns rather than understand meaning, interpreting these patterns as evidence of understanding is a misattribution. This misconception is extended by the notion that generative models can 'explain their reasoning.' Anecdotally, these misconceptions which mistake the form of language for its meaning are driving cognitive deference to AI systems in a way that displaces human agency. Such mistakes involve attributions of cognitive properties to the system, allowing them to be treated like intentional 'minds'. Importantly, perceived epistemic authority and superiority make that 'mind' worth deferring to. Anthropomorphism in tandem with epistemic authority and superiority, we argue, increases the risk that users generalise from narrow task performance to broader competencies, committing the "fallacy of composition". This paper explores misconceptions of Chain of Thought (CoT) reasoning as sufficient transparent 'explanation'. We explore how anthropomorphism acts as a vehicle for cognitive deference of decision-making and perceptions of epistemic superiority in AI systems. Finally, we explore CoT's ramifications using medicine as an example. Taken together, we conclude that explainability operationalised through CoT is highly problematic and undercuts the very ethical principles it is intended to serve. While CoT may be valuable for improving output accuracy and moderation, we suggest that the practice of asking models to provide user facing reasons for their outputs should be actively discouraged, and AI literacy training should explicitly caution against its use.
BibTeX
@misc{lack2026when,
  author = {Oliver Lack and Lana Tikhomirov and Humphrey Thompson and Carolyn Semmler and Melissa McCradden},
  title = {{When Reasoning looks like Hidden Understanding: How Chain-of-Thought Fosters Anthropomorphism, Perceptions of Epistemic Superiority, and Cognitive Deference}},
  year = {2026},
  eprint = {2611.13456},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13456},
}
Figure