Closing the Deference Gap: Steering Verifier Agents Toward What They Already Know

Multi-agent pipelines for code and math split a problem across roles. A generator writes a solution, and a separate verifier (a "Tester" or "Judge") checks it before the answer is accepted. The verifier exists because a single model thread tends not to catch its own mistakes. But a verifier built from the same model, reading the generator's confident rationale, can rubber-stamp a buggy solution. Told to be critical, it can instead reject correct work at random. We propose to measure, and then close, a specific failure we call the deference gap. This is the case where a verifier's internal activations already encode that a candidate is wrong, while its written verdict still says "looks good." The intervention is activation steering, meaning a small vector added to the model's hidden states during generation. The steering is conditional: it pushes the verifier's verdict toward its own internally read-out belief about correctness, and only as hard as that belief warrants.

The core claim is testable and falsifiable. On naturally occurring generator errors on HumanEval-ET, MBPP-ET and Level 5 MATH, three things should hold for open-weight 7–32B models. A linear probe on the verifier's hidden states will detect incorrect candidates more accurately than the verifier's own verdict does. This gap will widen when the verifier sees the generator's reasoning. Belief-conditioned steering of the verdict will recover most of the gap without raising false rejections. If there is no gap, the project reports a clean negative result about where verifier failures come from.

Why this, and why not the obvious combination

The obvious intersection is "replace role prompts with role vectors in a multi-agent pipeline." It is only a small step beyond existing work. Improving LLM Reasoning through Interpretable Role-Playing Steering already injects role features into a single model's residual stream, drawing them from sparse autoencoders (which split activations into interpretable features). It beats role prompting in 25 of 27 settings, and it lifts CSQA from 31.86% to 39.80% on Llama 3.1-8B. Swapping that into a Coder/Tester loop would mostly re-demonstrate that result. It would also inherit a known hazard: a static "be critical" push trades rubber-stamping for blind contrarianism.

Playing Devil's Advocate shows both sides of that hazard. A "Judge" persona vector reduced sycophancy while answering 14/16 factual probes correctly. A targeted anti-sycophancy vector built by contrastive activation addition dropped to 9/16, against an unsteered 12/16. That work motivates us, because the calibration of the steering direction clearly matters. It also constrains us, because it tested subjective philosophy questions put to one model by a human user. It did not test an automated verifier judging objectively checkable work.

Two further strands supply the method. Knowledge and decision states are linearly readable inside the model: Do I Know This Entity? finds directions for knowledge awareness, and Tool Calling is Linearly Readable and Steerable recovers 61–82% BFCL accuracy from hidden-state readouts, compared with 2–10% from the base models' direct generation. The second strand is closed-loop steering. Activation Steering with a Feedback Controller shows that fixed-strength steering overshoots and that steering strength should respond to a measured error.

We combine these into a steering signal that is conditioned on the model's own belief, rather than a fixed persona. The multi-agent setting is where this matters. There the verifier's context is deliberately separated from the generator's, and the tool-calling paper's warning about steering becoming unstable over long multi-turn histories is least likely to apply.

Core Claim and Mechanism

Consider the verifier's hidden state $h$ at a middle layer $l$, taken at the last token of the candidate solution. We fit two directions on held-out data.

The first is a correctness probe $w_c$, a logistic readout trained to predict whether the candidate passes the extended tests. It gives $p = \sigma(w_c^\top h)$, where $\sigma$ is the logistic function.

The second is a verdict direction $w_v$, the difference in mean activations between episodes where the verifier went on to write "accept" and episodes where it wrote "reject." This follows the construction in Steering Llama 2 via Contrastive Activation Addition.

During generation of the verdict and critique, we then apply

$$h' = h + \alpha\,(2p - 1)\,\frac{w_v}{\lVert w_v\rVert},$$

with a single gain $\alpha$ tuned on a validation split. The steering pushes toward accept when the model internally believes the candidate is correct, and toward reject when it does not. Its strength is proportional to the model's confidence, so it goes quiet near $p=0.5$. That is the sense in which it avoids a uniform skeptic push.

The deference gap itself is simply the AUROC of $p$ minus the AUROC of the unsteered verbal verdict, both measured on the same candidates. AUROC is the area under the ROC curve, the probability that a correct candidate is scored above an incorrect one.

Research Program

Build the testbed. The first phase builds a generator–verifier–repair pipeline in the style of Self-collaboration Code Generation via ChatGPT, with Analyst, Coder and Tester roles, and a Thinker/Judge variant for math in the style of MACM. The pipeline runs on Llama 3.1-8B, Qwen 2.5-7B, Gemma 2-9B and Qwen 3-32B.

Candidates come from the generator's own sampled outputs. Code candidates are labeled by the HumanEval-ET and MBPP-ET edge-case tests, and math candidates by answer match on MATH Level 5. The bugs are therefore natural errors rather than planted ones. Each candidate is shown to the verifier under three exposure conditions: code or answer only; plus the generator's comments and rationale; and plus the full shared message history.

Measure the gap. The second phase measures the deference gap per model, layer and exposure condition. It asks two questions. Does the gap exist at all? Does it grow as the verifier sees more of the generator's framing?

Close the gap. The third phase applies the conditional steering above. It compares the steering against static persona vectors and against simpler non-steering fixes, then studies what the steered critique contributes to the repair round.

Test transfer. The fourth phase tests whether directions fitted on code transfer to math verification without refitting. Transfer would indicate a general "suspects an error" representation rather than a dataset artifact.

Baselines

Naive reference points. The weakest reference is no verifier: the generator's first answer is taken as final. It sets the floor, and it tells us how much any verifier helps on these models. The second naive reference is the prompted Tester or Judge exactly as described in Self-collaboration and MACM, reimplemented on our open models with identical prompts, sampling settings and number of repair rounds.

Existing numbers. The file gives numbers for this family of pipelines only on proprietary models. Self-collaboration reports Pass@1 gains of 29.9% on HumanEval over plain ChatGPT, up to 34.6% on the ET variants, and 90.2% HumanEval Pass@1 with GPT-4. MACM reports Level 5 MATH rising from 54.68% to 76.73% on GPT-4 Turbo. For Llama 3.1-8B, Qwen 2.5-7B or Gemma 2-9B running these pipelines, we found no reported result on Lacuna, so we will establish those numbers ourselves. Beating the prompted verifier means a higher bug-catch rate at a matched false-rejection rate, and a higher final Pass@1 after the same number of repair rounds.

Strongest steering competitor. This is the static Judge or Skeptic persona vector of Playing Devil's Advocate. It is the strongest because it is the only existing method shown to reduce sycophancy while preserving factual accuracy. We extract it with that paper's recipe on our models, inject it at the same layers, and give it the same budget for sweeping its gain on the validation split.

Contrastive anti-sycophancy vector. We also include a targeted contrastive-activation-addition anti-sycophancy vector, fit on verifier accept/reject contrasts. It is the natural "just steer toward reject when wrong" alternative, and it is expected to over-reject, as it did on the 16 factual probes.

Ablation of the conditioning. The key ablation removes the conditioning on belief: the same $w_v$ direction with a constant sign and gain. If constant-sign steering matches the conditional version, then the probe-conditioned signal is not doing the work, and the claim fails.

Fair comparison. Every method uses the same candidates, splits, layers searched and number of gain values tried. Methods are compared on the full curve of bug-catch rate against false-rejection rate, not at a single operating point.

Why not just…?

Why not just write a sharper verifier prompt, or hide the generator's rationale from the verifier? These are the most serious threats, and both are baselines here. The file reports that prompted role adherence collapses below 13B parameters, according to Self-collaboration. It also reports that instruction compliance falls to 15.6% in long-context stress tests in Agent Primitives. Neither result concerns verifier deference specifically.

We will run three prompt-side controls with the same tuning budget: a skeptic prompt, a firewalled verifier that sees stripped code with no comments or rationale, and a paraphrase-normalized candidate. If firewalling alone closes the gap, the gap was caused by surface framing, and the right recommendation is to firewall the verifier rather than steer it. We will report that result as the answer.

Why not just train a verifier LoRA (a small low-rank adapter fine-tuned on top of the base weights)? We will train one on exactly the labeled candidates used to fit $w_c$ and $w_v$. That keeps the supervision identical, which matters because steering is not given more labels. Taxonomy, Opportunities, and Challenges of Representation Engineering argues that fine-tuning carries compute costs and a general-capability tax, but that is an argument, not evidence on this task. The decisive comparison is out-of-domain transfer, from code to math. There, a two-vector intervention that reuses the model's existing representation may generalize better than an adapter fit to one domain. If the LoRA wins in-domain and ties on transfer, we will say so.

Why not just use the probe as the verifier and skip steering? If $p$ already beats the verbal verdict, then gating on $p$ with a re-prompt on low scores is simpler. Our bet is that the steered verifier's written critique carries information about what is wrong, and that this information improves the repair round beyond what a single accept/reject bit can provide. The test is final Pass@1 after one repair round under probe-gated re-prompting versus steered critique, holding the number of calls fixed. If they tie, the contribution shrinks to the diagnostic finding plus a probe-gated pipeline, which is still useful but smaller.

Proposed Experiments and Decisive Checks

The first decisive check is existence. We compare probe AUROC with verdict AUROC on held-out candidates, with bootstrap confidence intervals, for each model and exposure condition. We expect the gap to be positive and to widen from code-only to full-history exposure. A gap indistinguishable from zero ends the steering line of the work.

The second check is calibrated closure. We compare full bug-catch versus false-rejection curves for conditional steering, constant-sign steering, persona vectors, anti-sycophancy vectors, prompt controls and the LoRA. Success means conditional steering dominates constant-sign steering and the persona vector across the curve, and recovers most of the measured gap.

The third check is downstream value. We measure final Pass@1 on HumanEval-ET and MBPP-ET and accuracy on MATH Level 5 after a fixed number of repair rounds, alongside total tokens.

The fourth check is fluency and general utility. We run MMLU through TinyBenchmarks with and without steering, following the practice of the feedback-controller paper, to confirm that steering does not damage the model's general ability.

The fifth check is code-to-math transfer of $w_c$ and $w_v$ with no refitting. It separates a general "suspects an error" representation from a benchmark artifact.