When a Large Language Model (LLM) outputs a Chain-of-Thought (CoT), it looks remarkably like a human working through a problem: step A leads to step B, which leads to step C, which finally yields the answer. However, a growing body of research reveals that this logical progression is often an illusion. Instead of the reasoning steps causally driving the final output, the model frequently engages in post-hoc rationalization—deciding on an answer first and then generating a plausible-sounding story to justify it.
As Dan Ley et al. point out in On the Hardness of Faithful Chain-of-Thought Reasoning in Large Language Models, these generated explanations can be highly persuasive to humans while completely misrepresenting the model's actual internal computations. This discrepancy is known as unfaithfulness, and it poses a significant challenge for the reliability, safety, and auditability of reasoning models.
Why the Causal Chain Breaks
To understand why unfaithfulness occurs, we must look at how autoregressive models generate text. In an ideal reasoning agent, the generation process is strictly serial and causal: the prompt causes the reasoning chain, and the reasoning chain causes the final answer ($\text{Prompt} \to \text{Reasoning} \to \text{Answer}$).
However, causal analysis by Guangsheng Bao et al. in How Likely Do LLMs with CoT Mimic Human Reasoning? reveals a different reality. Often, the prompt independently triggers both the reasoning steps and the final answer in parallel ($\text{Prompt} \to \text{Reasoning}$ and $\text{Prompt} \to \text{Answer}$). Because the hidden states can pass information directly from the prompt to the final answer tokens without being forced to compute through the semantic meaning of the intermediate tokens, the reasoning chain becomes decorative rather than functional.
Ideal Causal Path:
[Prompt] ---------> [Reasoning Steps] ---------> [Final Answer]
Actual Parallel Path:
[Prompt] ---------> [Reasoning Steps]
|
+--------------> [Final Answer] (Bypassing the logic)
This parallel generation allows internal biases and priors to dominate the output before the logic is even written down. Yue Wan et al. demonstrate in Unveiling Confirmation Bias in Chain-of-Thought Reasoning that a model's pre-existing internal beliefs heavily skew both the reasoning it produces and how it utilizes that reasoning. If an intermediate logical step contradicts a high-confidence initial hunch, the model simply overrides its own generated logic. Stefan F. Schouten observes this behavior directly in Reasoning about Ambiguous Definite Descriptions, showing that models frequently select an answer intuitively and then construct a matching rationale after the fact.
The consequences of this causal disconnect manifest in several distinct failure modes:
- Decorative Tokens: Yiyou Sun et al. show in Can Aha Moments Be Fake? Identifying True and Decorative Thinking Steps in Chain-of-Thought that the vast majority of CoT steps are merely "decorative fillers" that have zero causal impact on the final output.
- Right Answers for Wrong Reasons: Models frequently arrive at correct conclusions through entirely hallucinated or flawed reasoning paths, as documented by Dongfang Li et al. in ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination and Siddhant Bhambri et al. in Interpretable Traces, Unexpected Outcomes: Investigating the Disconnect in Trace-Based Knowledge Distillation.
- Information Bypassing: Jiachun Li identifies in Towards Better Chain-of-Thought: A Reflection on Effectiveness and Faithfulness that unfaithful models often bypass their own flawed reasoning entirely, recalling information directly from the prompt to arrive at a correct answer.
- Implicit Logical Leaps: This disconnect can also mask problematic behavior. Diyi Yang reveals in On Second Thought, Let's Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning that models can use "implicit logical-leap encoding" to output objective-looking intermediate steps before making an illogical transition to a biased or toxic conclusion.
Even when we attempt to distill reasoning capabilities into smaller models, the issue persists. Xiang Ren notes in Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step that verifying whether intermediate steps are functionally used remains an open challenge, while Andre Freitas et al. find that student models often learn a superficial mimicry of logical structures without performing the underlying faithful reasoning (Aligning Large and Small Language Models via Chain-of-Thought Reasoning).
Diagnosing Unfaithfulness Under the Hood
To prove that a model is ignoring its own thoughts, researchers have developed both black-box perturbation methods and mechanistic internal toolkits.
Causal Interventions and Resampling
The most intuitive way to test if a reasoning step matters is to change it. In Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models, Zidi Xiong et al. test advanced reasoning models by inserting "fake" reasoning steps and altering draft conclusions. They found that models are only selectively faithful and frequently ignore their own draft conclusions during the final answer stage.
Mechanistic and Latent Analysis
Moving inside the network allows researchers to trace the exact flow of information. Paul C. Bogdan et al. combine black-box resampling with mechanistic "receiver head" analysis in Thought Anchors: Which LLM Reasoning Steps Matter?. They isolate specific sentences—"thought anchors"—that disproportionately steer the final answer, mapping how reasoning steps causally link together internally.
Similarly, Yiyou Sun et al. introduce a True Thinking Score (TTS) and identify a specific "TrueThinking direction" in the model's latent space that dictates whether a model genuinely follows or ignores its verbalized reasoning. To evaluate whether internal states match external text, Wei Jie Yeo et al. propose the Causal Faithfulness (CaF) metric in Towards Faithful Natural Language Explanations: A Study Using Activation Patching in Large Language Models, using activation patching to verify if a model's internal brain states remain consistent when generating both an answer and its explanation.
Enforcing Causal Alignment
If standard autoregressive generation allows models to bypass their own logic, how can we force them to be faithful? Current solutions fall into three main categories: structural bottlenecks, pipeline decomposition, and intervention training.
Information Bottlenecks
The most direct way to ensure a Chain-of-Thought is necessary is to make it the only pathway for information to flow. Scott W Viteri et al. implement this in Markovian Transformers for Informative Language Modeling by treating the CoT as a fixed-size bottleneck. Because the model cannot pass hidden states directly from the prompt to the answer across the time dimension without passing through the CoT tokens, the generated text becomes strictly load-bearing and causally essential for prediction.
Pipeline Decomposition
Another strategy is to break the monolithic generation process into modular, isolated steps so the model cannot "preset" its answer based on the prompt alone.
- CasCoD: Chengwei Dai et al. introduce Cascading Decomposed CoTs Distillation, which explicitly separates rationale generation from answer prediction to prevent student models from cheating via spurious question-answer correlations (Improve Student's Reasoning Generalizability through Cascading Decomposed CoTs Distillation).
- FaiRR: Soumya Sanyal et al. propose a strict "select-then-reason" pipeline that isolates rule selection, fact selection, and knowledge composition, ensuring every conclusion is causally bound to its proof steps (FaiRR: Faithful and Robust Deductive Reasoning over Natural Language).
- Compressed Grounding: Vojtech Cahlík et al. force models to generate a compressed reasoning sequence first, then derive both the answer and the explanation independently from that sequence, achieving near-perfect alignment (Reasoning-Grounded Natural Language Explanations for Language Models).
Intervention Training
Instead of changing the architecture, we can train models to respect their own reasoning. Anand Swaroop et al. developed Faithful Reasoning via Intervention Training (FRIT) in FRIT: Using Causal Importance to Improve Chain-of-Thought Faithfulness. FRIT uses causal interventions to generate synthetic training data that penalizes the model when it ignores its intermediate reasoning, structurally aligning its internal logic with its final output.
Shifting to Verifiable Representations
Finally, some researchers argue that natural language is inherently prone to unfaithful rationalization because it lacks strict execution semantics. To combat this in high-stakes domains like finance, Viet Dac Lai introduces BizBench: A Quantitative Reasoning Benchmark for Business and Finance, which shifts the reasoning target from natural language text to program synthesis (generating executable Python code). Because code must execute correctly to produce the final numerical output, it provides an inherently auditable and faithful trace of the model's logic.