TL;DR
Just because you can "read" a model's mind doesn't mean you can "steer" it. This paper demonstrates that while reasoning correctness is highly decodable from a model's internal activations, intervening along those "correctness directions" fails to consistently improve performance, challenging the sufficiency of the Linear Representation Hypothesis for complex reasoning tasks.
What Problem Does This Paper Address?
In the world of AI interpretability, the Linear Representation Hypothesis suggests that high-level concepts (like "truthfulness" or "sentiment") are represented as specific directions in a model's activation space. If this holds, we should be able to find a "correctness vector" and nudge the model toward it to improve its reasoning.
However, reasoning is dynamic. A model might be "on the right track" at one step and hit a dead end the next. The authors investigate whether these internal signals of correctness actually function as causal levers. Can we identify a direction that represents "being on the right track" and use it to steer a model toward a correct solution?
Methods
To study this without the noise of billion-parameter models, the authors built a "glass-box" environment:
- ChainSearch: A controlled arithmetic task based on the game "Countdown." Unlike standard tasks, it includes explicit backtracking (using a
BACKtoken). This makes the model's internal search process visible and measurable. - The Model: A compact, 3-million-parameter transformer trained from scratch. Its size allows for exhaustive probing of every layer, head, and component.
- Direction Finding: They used three methods to find "correctness directions":
- Linear Probes: Training a classifier to predict if a state is "reachable" (solvable) or if the final answer will be correct.
- Contrastive Activation Addition (CAA): Taking the difference between mean activations of "correct" vs. "incorrect" reasoning paths.
- Direct Optimization: Freezing the model and mathematically optimizing a vector to maximize the probability of a correct final answer.
- Causal Steering: They applied these vectors during the model's generation process to see if they actually improved the success rate.
Datasets
The authors introduced ChainSearch, a dataset of arithmetic reasoning traces.
- Scale: 1,016,676 SFT (Supervised Fine-Tuning) examples and nearly 7 million exploratory traces for pretraining.
- Complexity: It uses a six-input version of Countdown, which is NP-complete and has a state space 5,400 times larger than the common "Game of 24" benchmark.
- Disjoint Splitting: To prevent the model from simply memorizing solutions, the authors ensured that the training and test sets never shared the same target numbers.
Key Insights and Ideas
- Decodability $\neq$ Causality: This is the central "aha" moment. Linear probes could predict whether a model would eventually get the answer right with an AUROC of ~0.94 (see Figure 2). However, steering the model along these highly predictive directions did almost nothing to improve actual performance.
- Stability is Component-Dependent: Correctness directions are most stable in attention heads compared to MLPs or residual streams. Even so, stability didn't translate to causal utility.
- The "Correctness" Mirage: A direction might look like it captures "correctness" along one dimension (e.g., across different time steps) but fail entirely when tested against another (e.g., across different layers).
Key Results
- High Decodability: As shown in Figure 2, the model's internal state becomes increasingly transparent regarding its eventual success as generation progresses.
- Failed Steering: In causal tests, steering rarely produced a consistent benefit. For example, using CAA directions improved only 5 out of 256 test cases while simultaneously worsening 4 others—essentially a net-zero gain.
- Direct Optimization Failure: Even vectors explicitly optimized to increase the likelihood of a correct answer failed to generalize when the model was allowed to generate a new, fresh response.
- Attention Head Advantage: Attention heads showed the highest "Successive-position stability" ($S^S$), meaning the correctness direction stayed relatively consistent as the model moved from one token to the next (see Figure 3).
Limitations and Future Work
- Model Scale: The study uses a 3M parameter model. While this allows for "rigorous search," it remains to be seen if massive LLMs exhibit different causal dynamics.
- Task Specificity: ChainSearch is a synthetic arithmetic task. The geometry of "correctness" in natural language reasoning (like law or creative writing) might be more complex or non-linear.
- Non-linear Representations: The failure of linear steering suggests that reasoning correctness might be encoded in non-linear manifolds that simple additive vectors cannot capture. The authors suggest future work should focus on these non-linear geometries.
Summary updated