The Causal Disconnect in Language Model Reasoning

2,103 concepts1,293 papers50 proposals

When a Large Language Model (LLM) outputs a Chain-of-Thought (CoT), it looks remarkably like a human working through a problem: step A leads to step B, which leads to step C, which finally yields the answer. However, a growing body of research reveals that this logical progression is often an illusion. Instead of the reasoning steps causally driving the final output, the model frequently engages in post-hoc rationalization—deciding on an answer first and then generating a plausible-sounding story to justify it.

As Dan Ley et al. point out in On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models, these generated explanations can be highly persuasive to humans while completely misrepresenting the model's actual internal computations. This discrepancy is known as unfaithfulness, and it poses a significant challenge for the reliability, safety, and auditability of reasoning models.

Why the Causal Chain Breaks

To understand why unfaithfulness occurs, we must look at how autoregressive models generate text. In an ideal reasoning agent, the generation process is strictly serial and causal: the prompt causes the reasoning chain, and the reasoning chain causes the final answer ($\text{Prompt} \to \text{Reasoning} \to \text{Answer}$).

However, causal analysis by Guangsheng Bao et al. in How Likely Do LLMs with CoT Mimic Human Reasoning? reveals a different reality. Often, the prompt independently triggers both the reasoning steps and the final answer in parallel ($\text{Prompt} \to \text{Reasoning}$ and $\text{Prompt} \to \text{Answer}$). Because the hidden states can pass information directly from the prompt to the final answer tokens without being forced to compute through the semantic meaning of the intermediate tokens, the reasoning chain becomes decorative rather than functional.

Ideal Causal Path:
[Prompt] ---------> [Reasoning Steps] ---------> [Final Answer]

Actual Parallel Path:
[Prompt] ---------> [Reasoning Steps]
   |
   +--------------> [Final Answer] (Bypassing the logic)

This parallel generation allows internal biases and priors to dominate the output before the logic is even written down. Yue Wan et al. demonstrate in Unveiling Confirmation Bias in Chain-of-Thought Reasoning that a model's pre-existing internal beliefs heavily skew both the reasoning it produces and how it utilizes that reasoning. If an intermediate logical step contradicts a high-confidence initial hunch, the model simply overrides its own generated logic. Stefan F. Schouten observes this behavior directly in Reasoning about Ambiguous Definite Descriptions, showing that models frequently select an answer intuitively and then construct a matching rationale after the fact.

The consequences of this causal disconnect manifest in several distinct failure modes:

Even when we attempt to distill reasoning capabilities into smaller models, the issue persists. Xiang Ren notes in Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step that verifying whether intermediate steps are functionally used remains an open challenge, while Andre Freitas et al. find that student models often learn a superficial mimicry of logical structures without performing the underlying faithful reasoning (Aligning Large and Small Language Models via Chain-of-Thought Reasoning).

Diagnosing Unfaithfulness Under the Hood

To prove that a model is ignoring its own thoughts, researchers have developed both black-box perturbation methods and mechanistic internal toolkits.

Causal Interventions and Resampling

The most intuitive way to test if a reasoning step matters is to change it. In Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models, Zidi Xiong et al. test advanced reasoning models by inserting "fake" reasoning steps and altering draft conclusions. They found that models are only selectively faithful and frequently ignore their own draft conclusions during the final answer stage.

Mechanistic and Latent Analysis

Moving inside the network allows researchers to trace the exact flow of information. Paul C. Bogdan et al. combine black-box resampling with mechanistic "receiver head" analysis in Thought Anchors: Which LLM Reasoning Steps Matter?. They isolate specific sentences—"thought anchors"—that disproportionately steer the final answer, mapping how reasoning steps causally link together internally.

Similarly, Yiyou Sun et al. introduce a True Thinking Score (TTS) and identify a specific "TrueThinking direction" in the model's latent space that dictates whether a model genuinely follows or ignores its verbalized reasoning. To evaluate whether internal states match external text, Wei Jie Yeo et al. propose the Causal Faithfulness (CaF) metric in Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models, using activation patching to verify if a model's internal brain states remain consistent when generating both an answer and its explanation.

Enforcing Causal Alignment

If standard autoregressive generation allows models to bypass their own logic, how can we force them to be faithful? Current solutions fall into three main categories: structural bottlenecks, pipeline decomposition, and intervention training.

Information Bottlenecks

The most direct way to ensure a Chain-of-Thought is necessary is to make it the only pathway for information to flow. Scott W Viteri et al. implement this in Markovian Transformers for Informative Language Modeling by treating the CoT as a fixed-size bottleneck. Because the model cannot pass hidden states directly from the prompt to the answer across the time dimension without passing through the CoT tokens, the generated text becomes strictly load-bearing and causally essential for prediction.

Pipeline Decomposition

Another strategy is to break the monolithic generation process into modular, isolated steps so the model cannot "preset" its answer based on the prompt alone.

Intervention Training

Instead of changing the architecture, we can train models to respect their own reasoning. Anand Swaroop et al. developed Faithful Reasoning via Intervention Training (FRIT) in FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness. FRIT uses causal interventions to generate synthetic training data that penalizes the model when it ignores its intermediate reasoning, structurally aligning its internal logic with its final output.

Shifting to Verifiable Representations

Finally, some researchers argue that natural language is inherently prone to unfaithful rationalization because it lacks strict execution semantics. To combat this in high-stakes domains like finance, Viet Dac Lai introduces BizBench: A Quantitative Reasoning Benchmark for Business and Finance, which shifts the reasoning target from natural language text to program synthesis (generating executable Python code). Because code must execute correctly to produce the final numerical output, it provides an inherently auditable and faithful trace of the model's logic.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_post_hoc_rationalization_c5c23599,
  title = {post-hoc rationalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-hoc-rationalization/rdr_c5c2359931b54cc3a0822885b9dd918a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_post_hoc_rationalization_c5c23599_v1,
  title = {post-hoc rationalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/post-hoc-rationalization/rdr_c5c2359931b54cc3a0822885b9dd918a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake