TL;DR
This paper provides the first feature-level causal investigation into whether Chain-of-Thought (CoT) prompting is truly "faithful" to an LLM's internal reasoning. By combining Sparse Autoencoders (SAEs) and activation patching, the authors discover that CoT prompting induces sparser, more interpretable, and causally effective internal representations—but only once a model crosses a critical scale threshold (observed in Pythia-2.8B but not Pythia-70M).
What Problem Does This Paper Address?
While Chain-of-Thought (CoT) prompting dramatically improves Large Language Model (LLM) accuracy on complex tasks, a fundamental question remains: Is CoT faithful? In other words, do the intermediate "thoughts" generated on the surface actually reflect the model's internal problem-solving process, or are they just plausible-sounding scaffolding?
Previous attempts to look inside the LLM "black box" have been limited. Coarse layer- or component-level interventions cannot pinpoint specific concepts, while neuron-level analyses suffer from polysemanticity (where a single neuron fires for multiple unrelated concepts) and superposition. This paper addresses this gap by extracting clean, monosemantic features to causally test how CoT alters internal reasoning circuits.
Methods
The authors introduce a feature-level causal intervention framework that integrates two powerful mechanistic interpretability tools:
- Sparse Autoencoders (SAEs): The authors trained separate SAEs on the residual stream activations (at layer 2, final token position) under both CoT and NoCoT conditions. The SAEs disentangle dense, overlapping representations into a sparse set of human-interpretable features.
- Activation Patching: To test causality, they performed a feature-swap intervention. They took sparse features from a CoT run and patched them into a NoCoT run (and vice versa), measuring the exact change in the log-probability of the correct answer ($\Delta \log P$).
To understand how these features accumulate, they introduced Patch-Curve Analysis, comparing the gradual injection of the most highly differentiated features ("Top-K") against a baseline of randomly sampled features ("Random-K"). They also used GPT-3.5-turbo to automatically generate natural language explanations and interpretability scores for the extracted features.
Datasets
The framework was evaluated using GSM8K (Grade School Math 8K), a standard and challenging benchmark consisting of multi-step math word problems. The authors utilized up to 1,000 problem pairs from the training split to compare models prompted with CoT (few-shot examples with step-by-step solutions) against a NoCoT baseline (the question alone).
Key Insights and Ideas
- Faithfulness Requires Scale: CoT is not an inherent property of all language models; it requires a minimum capacity threshold. In the 70M parameter model, transferring CoT features into NoCoT runs disrupted inference. In contrast, the 2.8B model successfully internalized the structured prompting, showing a robust, positive causal transfer.
- CoT Induces "Structured Sparsity": CoT prompting fundamentally reshapes the model's internal activation patterns. Instead of firing broadly, the model suppresses the majority of its neurons to near-zero and strategically concentrates resources into a highly selective subset of features. This serves as an unsupervised mitigation of the superposition problem.
- Causal Signals are Distributed, Not Concentrated: In the 2.8B model, randomly patching CoT-activated features consistently outperformed patching only the "Top-K" highest-activating features. This reveals an "aha moment": crucial reasoning information is widely distributed across many moderately activated features rather than being locked behind a few dominant directions. Selecting only the top peaks risks overfitting to local signals.
Key Results
- Causal Impact: Swapping CoT features into NoCoT trajectories significantly boosted the 2.8B model's confidence in generating the correct answer, improving the log-probability from 1.2 to 4.3.
- Scale Disparity: In Pythia-2.8B, the log-probability changes after CoT patching were overwhelmingly positive. In Pythia-70M, the effects were highly volatile and often symmetric around zero, indicating that feature transfer was unstable or disruptive.
- Interpretability Boost: Features activated by CoT prompts in the 2.8B model achieved significantly higher semantic explanation scores (mean of 0.056) compared to NoCoT features (mean of -0.013, $p = 0.004$). For the 70M model, CoT provided no statistically significant improvement in feature interpretability ($p = 0.935$).
Limitations and Future Work
While groundbreaking, the authors note several limitations:
- Static Snapshot: The activation patching only targets the residual stream of the final token and does not dynamically trace how causal effects evolve token-by-token throughout the entire generated text chain.
- Model Size Bounds: The study caps out at Pythia-2.8B. Further validation is required on larger, state-of-the-art models (like LLaMA-7B or larger) to ensure these structural properties hold true at massive scales.
- Indirect Explanations: The semantic interpretation module relies entirely on automated LLM scoring, which provides an external proxy rather than grounding explanations directly in specific attention heads or circuits.
Future Directions: The authors suggest moving toward token-level and path-based causal tracing (such as path patching) combined with SAE feature decomposition to map out complete, sparse causal circuits.
Summary updated