How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding

Xi Chen, Aske Plaat, Niki van Stein

TL;DR

This paper provides the first feature-level causal investigation into whether Chain-of-Thought (CoT) prompting is truly "faithful" to an LLM's internal reasoning. By combining Sparse Autoencoders (SAEs) and activation patching, the authors discover that CoT prompting induces sparser, more interpretable, and causally effective internal representations—but only once a model crosses a critical scale threshold (observed in Pythia-2.8B but not Pythia-70M).

What Problem Does This Paper Address?

While Chain-of-Thought (CoT) prompting dramatically improves Large Language Model (LLM) accuracy on complex tasks, a fundamental question remains: Is CoT faithful? In other words, do the intermediate "thoughts" generated on the surface actually reflect the model's internal problem-solving process, or are they just plausible-sounding scaffolding?

Previous attempts to look inside the LLM "black box" have been limited. Coarse layer- or component-level interventions cannot pinpoint specific concepts, while neuron-level analyses suffer from polysemanticity (where a single neuron fires for multiple unrelated concepts) and superposition. This paper addresses this gap by extracting clean, monosemantic features to causally test how CoT alters internal reasoning circuits.

Methods

The authors introduce a feature-level causal intervention framework that integrates two powerful mechanistic interpretability tools:

  1. Sparse Autoencoders (SAEs): The authors trained separate SAEs on the residual stream activations (at layer 2, final token position) under both CoT and NoCoT conditions. The SAEs disentangle dense, overlapping representations into a sparse set of human-interpretable features.
  2. Activation Patching: To test causality, they performed a feature-swap intervention. They took sparse features from a CoT run and patched them into a NoCoT run (and vice versa), measuring the exact change in the log-probability of the correct answer ($\Delta \log P$).
Figure 1: Workflow of the approach: After SAE, we do Activation patching, Feature Interpretation, and Activation Sparsity Analysis. All three confirm that CoT improves faithfulness of reasoning

To understand how these features accumulate, they introduced Patch-Curve Analysis, comparing the gradual injection of the most highly differentiated features ("Top-K") against a baseline of randomly sampled features ("Random-K"). They also used GPT-3.5-turbo to automatically generate natural language explanations and interpretability scores for the extracted features.

Datasets

The framework was evaluated using GSM8K (Grade School Math 8K), a standard and challenging benchmark consisting of multi-step math word problems. The authors utilized up to 1,000 problem pairs from the training split to compare models prompted with CoT (few-shot examples with step-by-step solutions) against a NoCoT baseline (the question alone).

Key Insights and Ideas

  • Faithfulness Requires Scale: CoT is not an inherent property of all language models; it requires a minimum capacity threshold. In the 70M parameter model, transferring CoT features into NoCoT runs disrupted inference. In contrast, the 2.8B model successfully internalized the structured prompting, showing a robust, positive causal transfer.
  • CoT Induces "Structured Sparsity": CoT prompting fundamentally reshapes the model's internal activation patterns. Instead of firing broadly, the model suppresses the majority of its neurons to near-zero and strategically concentrates resources into a highly selective subset of features. This serves as an unsupervised mitigation of the superposition problem.
  • Causal Signals are Distributed, Not Concentrated: In the 2.8B model, randomly patching CoT-activated features consistently outperformed patching only the "Top-K" highest-activating features. This reveals an "aha moment": crucial reasoning information is widely distributed across many moderately activated features rather than being locked behind a few dominant directions. Selecting only the top peaks risks overfitting to local signals.

Key Results

  • Causal Impact: Swapping CoT features into NoCoT trajectories significantly boosted the 2.8B model's confidence in generating the correct answer, improving the log-probability from 1.2 to 4.3.
  • Scale Disparity: In Pythia-2.8B, the log-probability changes after CoT patching were overwhelmingly positive. In Pythia-70M, the effects were highly volatile and often symmetric around zero, indicating that feature transfer was unstable or disruptive.
Figure 3: Distribution of log-probability changes after patching the top 20 CoT features into NoCoT runs under dictionary ratio 4. Left: Pythia-70M; Right: Pythia-2.8B. While 2.8B shows a strong positive shift indicating consistent benefit from CoT features, 70M shows highly variable effects, including large performance drops, suggesting unstable or less effective feature transfer.
  • Interpretability Boost: Features activated by CoT prompts in the 2.8B model achieved significantly higher semantic explanation scores (mean of 0.056) compared to NoCoT features (mean of -0.013, $p = 0.004$). For the 70M model, CoT provided no statistically significant improvement in feature interpretability ($p = 0.935$).
Figure 2: Comparison of feature explanation scores under CoT and NoCoT prompts. Left: Pythia-70M; Right: Pythia-2.8B. The 2.8B model shows higher explanation scores under CoT, indicating stronger causal features are learned in the larger model when CoT prompting is applied. Each plot is based on 50 features per condi- tion.

Limitations and Future Work

While groundbreaking, the authors note several limitations:

  • Static Snapshot: The activation patching only targets the residual stream of the final token and does not dynamically trace how causal effects evolve token-by-token throughout the entire generated text chain.
  • Model Size Bounds: The study caps out at Pythia-2.8B. Further validation is required on larger, state-of-the-art models (like LLaMA-7B or larger) to ensure these structural properties hold true at massive scales.
  • Indirect Explanations: The semantic interpretation module relies entirely on automated LLM scoring, which provides an external proxy rather than grounding explanations directly in specific attention heads or circuits.

Future Directions: The authors suggest moving toward token-level and path-based causal tracing (such as path patching) combined with SAE feature decomposition to map out complete, sparse causal circuits.

Summary updated

Abstract

Chain-of-thought (CoT) prompting boosts Large Language Models accuracy on multi-step tasks, yet whether the generated "thoughts" reflect the true internal reasoning process is unresolved. We present the first feature-level causal study of CoT faithfulness. Combining sparse autoencoders with activation patching, we extract monosemantic features from Pythia-70M and Pythia-2.8B while they tackle GSM8K math problems under CoT and plain (noCoT) prompting. Swapping a small set of CoT-reasoning features into a noCoT run raises answer log-probabilities significantly in the 2.8B model, but has no reliable effect in 70M, revealing a clear scale threshold. CoT also leads to significantly higher activation sparsity and feature interpretability scores in the larger model, signalling more modular internal computation. For example, the model's confidence in generating correct answers improves from 1.2 to 4.3. We introduce patch-curves and random-feature patching baselines, showing that useful CoT information is not only present in the top-K patches but widely distributed. Overall, our results indicate that CoT can induce more interpretable internal structures in high-capacity LLMs, validating its role as a structured prompting method.
BibTeX
@inproceedings{chen2026how,
  author = {Xi Chen and Aske Plaat and Niki van Stein},
  title = {{How does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i36.40281},
}
Figure