Reading Step Errors Out of a Verifier's Hidden State, Trained Only on Final Answers

When a language model checks a math solution, it may already "know" which line is wrong. That knowledge would sit in its internal activations long before it writes any critique. This note proposes to test that idea directly. We train a single linear readout on a frozen model's mid-layer activations to score each step of a solution. The readout is trained only with final-answer correctness labels, never with step labels. We then ask two questions. Does this cheap readout find the first wrong step as well as fully fine-tuned process verifiers? And does it depend on the model being put into an active "checking" mode rather than passively reading? If both hold, step-level supervision for a new domain costs one forward pass per solution and a logistic regression. Hiring annotators to label 800,000 steps, as was done for Let's Verify Step by Step, would no longer be needed.

The second question comes from systems neuroscience. In gerbils discriminating slow from fast sound modulations, a linear decoder over a few hundred parietal neurons reaches about 80% accuracy, matching the animal's own discrimination performance. It does so only when the animal is performing the task. Under passive listening to identical sounds, decoding is significantly degraded, and the population state "untangles" into separable choice clusters only over the first 600 ms of engaged listening (Transformation of acoustic information to sensory decision variables in the parietal cortex). Our hypothesis is that verifier models behave the same way. Information about whether a step is sound becomes linearly readable from the model's internal state when the model is framed as checking the solution. When the model merely continues or reads it, that information stays entangled. We also expect the effect to be measurable with the same geometric tools neuroscientists use.

Core Claim

The claim has three parts. First, a linear head on frozen mid-layer states, trained with the weakest-link outcome loss of Learning Discriminative Process Reward Models without Step Labels, localizes first errors on ProcessBench. Its accuracy should be competitive with prompted critics at far lower inference cost and should clearly exceed training-free confidence signals. Second, prepending a verification framing (the "task engagement" manipulation) measurably improves both the readout's accuracy and the geometric separability of sound-step versus flawed-step states. That separability is quantified by manifold capacity, radius and dimension in the sense used by Yao et al. and by Toward a computational theory of manifold untangling. Third, the layer where separability peaks can be chosen from outcome-labeled data alone. This addresses the extreme layer sensitivity reported for Latent Phase-Shift Rollback.

Position Relative to Prior Work

Three lines of work motivate and constrain this proposal.

The first supplies the training signal. Learning Discriminative Process Reward Models without Step Labels treats step quality as a hidden variable. A correct solution needs every step sound, so its step scores are aggregated with a minimum. An incorrect one needs at least one bad step, so its step scores are aggregated with a product. This matches step-supervised verifiers on MATH-500, AIME and ProcessBench, but it fine-tunes an entire model. We reuse the loss unchanged and replace the fine-tuned model with a linear map on frozen activations. Concretely, let $h_t$ be the layer-$\ell$ hidden state at the last token of step $t$, and let $s_t=\sigma(w^\top h_t+b)$ be the step score. Let $y\in\{0,1\}$ be the final-answer label. The loss is

$$ \mathcal{L} = -\,y\log \min_t s_t \;-\;(1-y)\log\Big(1-\prod_t s_t\Big), $$

and only $w$ and $b$ are learned. If a frozen linear readout suffices, the step-error information already exists in the base model. In that case Kim et al.'s gains come mainly from learning to read that information, not from learning new representations. If it does not suffice, the comparison tells us how much representation learning matters.

The second line provides label-free internal signals. Mind the Gap flags errors when the model's token-level confidence collapses, raising step-level localization accuracy on the ProcessBench GSM8K split from 27.66% to 46.11% for Qwen3-8B. Latent Phase-Shift Rollback detects sharp direction reversals in the residual stream around layer 14 of 32 and steers at layer 16. Both read a scalar summary of one stream, either output entropy or cosine change. The neuroscience literature motivates a different readout. Single neurons poorly predict choice, while small pooled ensembles track behavior (Neural Ensemble Codes for Stimulus Periodicity in Auditory Cortex). By analogy, a learned direction over the whole hidden state should beat a hand-chosen scalar. We test that analogy rather than assume it.

The third line is the geometry. Yao et al. report that engagement turns entangled stimulus representations into low-dimensional, linearly separable ones. The untangling theory paper argues instead for high-dimensional expansion. LPSR worries that large models spread information too thinly for low-rank steering. These accounts disagree on whether step verification benefits from expansion or compression. Our engagement manipulation turns that disagreement into a measurement. It tells us whether verification framing lowers or raises the dimension of the flawed-step manifold while increasing its separability.

The closest existing combination is Mind the Gap's "reasoning manifold." That work is training-free, uses only output entropy, and has no notion of an engaged versus passive condition. The proposal here differs in three ways. It trains a population readout from outcome labels. It manipulates engagement as an experimental variable. And it measures separability geometrically rather than inferring it from confidence.

Research Program

We build the readout for two open models that have reported ProcessBench-adjacent numbers in the file: DeepSeek-R1-Distill-Llama-8B and Qwen3-8B. Training data are solutions sampled by several generator models on MATH training problems, labeled only by whether the final answer matches. These solutions are read by the frozen verifier model under two conditions. In the passive condition, the model reads the solution as plain continuation text. In the engaged condition, the same tokens are preceded by an instruction to check each step for the first error. The activations at step boundaries are cached once per layer. Training the head is then a convex-ish problem over a few thousand dimensions and costs minutes.

The geometric analysis runs on held-out data with gold step labels (ProcessBench). At each layer and condition, we form two point clouds: states at sound steps and states at the first erroneous step. We compute linear separability, manifold capacity, radius and participation-ratio dimension. Layer selection for the deployed head uses only an outcome-labeled validation split, never the step-labeled data. This keeps the method label-free end to end. A further analysis splits ProcessBench by its four difficulty tiers. It tests whether readout quality degrades nonlinearly with difficulty, as choice-related neural covariation varies nonlinearly with stimulus strength in Stimulus-dependent relationships between behavioral choice and sensory neural responses.

Baselines

The naive reference point has two parts. One is chance-level first-error prediction on ProcessBench. The other is an outcome-only verifier that scores every step of a solution with its final-answer probability. The outcome-only verifier shows what is achievable with no localization ability at all. Any method must clearly beat it on the first-error F1 that ProcessBench reports, because ProcessBench finds that 51.8% of correct-answer Omni-MATH solutions contain invalid steps (ProcessBench).

The strongest label-free trained competitor is the fully fine-tuned discriminative verifier of Kim et al. We retrain it from the same base model on exactly the same outcome-labeled solutions with the same weakest-link loss. We give it the same hyperparameter search budget in trials, so the only difference is frozen-linear versus full fine-tuning. The file reports that it matches step-supervised verifiers but gives no ProcessBench F1. The file has no reported result on Lacuna for the discriminative PRM's ProcessBench score, so our retrained number is the reference. Beating it means matching its ProcessBench F1 within two points and its MATH-500 Best-of-N accuracy within one point while using orders of magnitude fewer trained parameters. Outright exceeding it is not expected and is not the claim.

The strongest prompted competitor is temporal-consistency self-checking (Temporal Consistency for LLM Reasoning Process Error Identification). It reports 67.2% on ProcessBench for DeepSeek-R1-Distill-Llama-8B, above GPT-4o's 62.5%. We run it with the authors' round settings on the same base model. We report both its score and its generated-token cost. A win for us means comparable F1 at a single forward pass with no generated tokens.

For the label-free internal signal, we rerun Evidence Drop on Qwen3-8B. The reported reference is 46.11% step-level localization accuracy on the GSM8K split (Mind the Gap). We also adapt LPSR's phase-shift detector as a pure detector at its reported layer. Beating these means higher localization accuracy on every ProcessBench split.

For context only, the file reports that step-supervised GPT-4 verification solved 78.2% of a MATH subset with Best-of-N (Let's Verify Step by Step), and o1-mini reaches 87.9 average F1 on ProcessBench. These are ceilings from much larger systems, not targets.

Two ablations isolate the proposal's ingredients. The first replaces the weakest-link loss with plain per-step regression onto the final label, which is an ordinary value probe. This tests whether the asymmetric aggregation is what produces localization. The second removes the engagement framing, using passive reading with an identical head and identical data. This tests the neuroscience-derived hypothesis directly.

Why not just…?

Why not just fine-tune the whole verifier, as Kim et al. do? This is the real threat. If full fine-tuning beats the frozen readout by a wide margin, the "free PRM" story fails. The retrained baseline above settles the question on identical data. Even in that case, the gap tells us how much step-error information the base model already encodes, which is useful when deciding whether a new domain needs a fine-tuned verifier at all. The cost argument is concrete. Retraining a linear head after each policy update is trivial, while refitting a full verifier is not. That matters because static step labels drift as the policy changes.

Why not just use output entropy, which needs no training at all? Entropy is a single scalar of the output distribution. The ensemble-coding results in neuroscience suggest that a pooled linear readout carries information no single unit does, and Mind the Gap's 46.11% localization on the easiest ProcessBench split leaves substantial room for improvement. The comparison is run on all four splits. If entropy matches the trained head, we will report that the training step is unnecessary.

Why not just prompt the model to critique the steps? Prompted self-correction cost Llama-3-8B nine points on MATH-500 in LPSR's experiments. Temporal consistency does work, at 67.2% on ProcessBench, but it multiplies generation cost across rounds. Our engagement condition uses the critic prompt but reads activations instead of generated text. The comparison therefore separates two possibilities: the prompt's benefit lies in the representation it induces, or it lies in the reasoning the model writes out.

Why not just train a value probe on outcome labels without the weakest-link structure? That probe is our first ablation. It is an honest competitor because it is even simpler. We expect it to rank solutions reasonably but localize poorly, since it has no pressure to concentrate blame on one step. If it localizes as well as the weakest-link head, the aggregation adds nothing in the frozen setting and we will say so.

Decisive Checks

Four results decide the idea. The first is the engagement test: on held-out ProcessBench, the engaged condition must raise both readout F1 and manifold capacity for flawed-versus-sound states relative to passive reading, at matched layers. If engagement leaves geometry unchanged, the neuroscience analogy is decorative and we drop it. The second is the layer test: the layer chosen from outcome-only validation must coincide within a few layers with the layer of peak gold-label separability. If it does not, label-free layer selection fails, and LPSR's layer-sensitivity problem remains unsolved. The third is transfer: ProcessBench solutions come from many generators, so a readout trained on one generator's solutions must hold up on others. The fourth is utility: on MATH-500, using the head to rerank Best-of-N samples from a fixed generator must beat majority voting and the outcome-only verifier at equal N. Alongside accuracy, we report the dimension of the flawed-step manifold under each condition. This settles, for verifier models at least, whether effective verification compresses or expands the representation.