Chain-of-Thought Hidden States Reveal Task-Solving Directions for Direct Answers

Houman Mehrafarin, Ioannis KonstasarXiv 2026 · arXiv

TL;DR

Large Language Models often "know" the correct answer internally long before they finish generating a lengthy Chain-of-Thought (CoT). By identifying a specific "task-solving direction" within the model's hidden states, researchers can steer a direct-answer prompt to achieve higher accuracy than CoT while using 90% fewer tokens.

What Problem Does This Paper Address?

Chain-of-Thought (CoT) prompting is the gold standard for complex reasoning in LLMs, but it has two major drawbacks: it is computationally expensive (generating hundreds of tokens) and it isn't foolproof—sometimes the model reasons its way to a wrong answer.

This paper investigates a fascinating discrepancy: the information needed to solve a task is often present in the model's internal hidden states even when the generated text is incorrect. The authors aim to understand if these "useful" internal states can be systematically identified and used to skip the long-winded reasoning chain entirely, moving straight to a correct direct answer.

Methods

The researchers employ a technique called activation patching to probe the model's internal representations.

  1. Labeling States: They run a model with a CoT prompt (the "source") and cache all hidden states. They then run the same question with a direct-answer prompt (the "target") and replace its final hidden state with one from the CoT run. If this "patch" causes the direct run to produce the correct answer, that CoT state is labeled recovering; otherwise, it is failed.
  2. Linear Probing: They train linear classifiers (probes) to see if these two classes of states are linearly separable in the model's representation space.
  3. Finding the Direction: For each question, they calculate a task-solving direction ($d$). This is essentially the vector pointing from the average of the "failed" states toward the average of the "recovering" states ($d \propto c^+ - \lambda c^-$).
  4. Spherical Intervention: To test this direction, they take a fresh direct-answer run and rotate its final hidden state toward the task-solving direction using Spherical Linear Interpolation (SLERP). This changes the direction of the state while preserving its original norm (magnitude).
Overview of the patching and rotation analysis on a multistep arithmetic question.

Datasets

The authors evaluated their approach across 25 tasks spanning six reasoning types: arithmetic, logical, temporal, spatial, causal, and linguistic.

  • GSM8K: Grade school math word problems.
  • StrategyQA: Questions requiring multi-step implicit reasoning.
  • BIG-Bench Hard (BBH): 23 subtasks including Boolean Expressions, Logical Deduction, and Geometric Shapes.
  • Models: Evaluations were performed on LLaMA 3.1 8B and Qwen 2.5 7B.

Key Insights and Ideas

  • Correctness is Linearly Decodable: The fact that a linear probe can distinguish "recovering" from "failed" states (with a mean AUROC of ~0.84) suggests that the model's internal "knowledge" of the correct path is structured and accessible, even if the output decoder misses it.
  • Directions are Question-Specific: A major finding is that there is no single "reasoning direction" for an entire task. The task-solving direction is specific to each individual question. Averaging these directions across a task results in a vector that fails to improve performance.
  • Generalization Across Reasoning Types: Probes trained on one task can often identify recovering states in other tasks, especially those of the same reasoning type (e.g., logical tasks generalize to other logical tasks). This suggests the model uses similar internal "circuits" for related types of problems.
Figure 2: Out-of-fold AUROC of each task’s probe, at layer 29 of LLaMA 3.1 8B and layer 25 of Qwen 2.5 7B.

Key Results

The intervention—rotating a single hidden state—produced dramatic improvements over both direct prompting and standard CoT:

  • Accuracy Boost: On LLaMA 3.1 8B, accuracy jumped from 46% (direct) to 85% (steered), significantly outperforming CoT (69%). Qwen 2.5 7B saw a similar rise from 52% to 86% (CoT: 77%).
  • Extreme Token Efficiency: The steered models reached these correct answers using an average of only 19 tokens, compared to the 197–245 tokens typically generated during a full CoT process.
  • Fixing "Broken" Reasoning: The method successfully recovered correct answers for 64-70% of the questions where both the direct prompt and the CoT prompt originally failed.

Limitations and Future Work

The primary limitation is that this method currently requires generating the CoT chain first to "find" the task-solving direction, making it an analysis tool rather than a drop-in replacement for inference.

However, the authors have created a valuable dataset of question-direction pairs. They suggest that future work could involve training a separate, smaller model (a regressor) to predict this task-solving direction directly from the input prompt. If successful, this would allow models to achieve CoT-level reasoning (or better) at the speed and cost of direct answering.

Summary updated

Abstract

Chain-of-Thought (CoT) prompting improves the performance of Large Language Models by eliciting intermediate reasoning steps. However, CoT is far from perfect: on some tasks, models perform worse than with direct-answer prompting, and on others they generate a few hundred tokens to reach the correct answer. We start by labelling each CoT hidden state as recovering if patching it into the direct prompt yields the correct answer, and failed otherwise. A linear probe shows that the model encodes these two classes differently. We then show that the representation space of the direct prompt holds a direction for each question that leads the model, in a short generation, to the correct answer more often than CoT does. Specifically, we rotate the direct prompt's last hidden state once, before generation, towards a direction built for each question from its own recovering and failed states. Across 25 tasks, these directions raise the direct prompt's mean accuracy from $46\%$ to $85\%$ on LLaMA 3.1 8B (CoT: $69\%$) and from $52\%$ to $86\%$ on Qwen 2.5 7B (CoT: $77\%$), with a tenth of CoT's tokens. CoT is thus not the shortest route to this accuracy, and these directions can be used in future work to train models to align their states with them.
BibTeX
@misc{mehrafarin2026chain,
  author = {Houman Mehrafarin and Ioannis Konstas},
  title = {{Chain-of-Thought Hidden States Reveal Task-Solving Directions for Direct Answers}},
  year = {2026},
  eprint = {2611.24598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24598},
}
Figure