Learning Domain Step Judges from an Agent's Own Inconsistency

When a language-model agent is run five times on the same clinical database task, it often succeeds once or twice and fails the rest. On the EHR-ChatQA benchmark, o4-mini solves 66.2% of adaptive-refinement tasks in at least one of five runs but only 7.5% in all five (From Conversation to Query Execution). The usual reading is that this inconsistency is a reliability problem to be measured. This proposal treats it as free supervision. A task that the agent sometimes solves and sometimes fails gives us matched pairs of attempts: same question, same documents, same tools, different outcomes. If both attempts are broken into "hops", meaning the facts, values, and tool calls the agent commits to one after another, the first hop where a failed run departs from every successful sibling is a strong candidate for the step that broke the task. Such labels come without clinician or domain-expert annotation, which is the main cost that blocks step-level supervision in specialized domains.

We propose to train a step-level judge on these divergence-anchored labels. The judge is a model that scores whether a proposed intermediate step is sound. We would deploy it at test time to choose among candidate next steps before they are executed. The goal is to raise the fraction of tasks an agent solves on every repeated attempt, not merely its best-case score, in domain settings where multi-hop reasoning over unfamiliar schemas and policies is the bottleneck. A colleague's two-sentence version would be: the runs where an agent succeeds on a task it usually fails show exactly which step matters, so we mine those contrasts to train a judge that picks the right step at that point. The tasks that generate this training signal are the same tasks that create the gap between best-case and every-time performance.

Core Claim

Let $p_t$ be the probability that the agent solves task $t$ in a single run. The two standard reliability metrics are then

$$ \text{Pass@}k = \mathbb{E}_t\!\left[1-(1-p_t)^k\right], \qquad \text{Pass}^k = \mathbb{E}_t\!\left[p_t^{\,k}\right], $$

where Pass@k counts a task as solved if any of $k$ runs succeeds and Pass$^k$ requires all $k$ to succeed. Tasks with $p_t$ near 0 or 1 contribute little to the difference between the two. The gap comes almost entirely from tasks with intermediate $p_t$, and these are exactly the tasks that produce both successful and failed sibling runs. The claim is that contrasting siblings at the level of committed facts localizes the failure-causing hop well enough to train a judge. Using that judge to select among candidate steps online should raise Pass$^k$ more than it raises Pass$^1$, which would narrow the gap.

This differs from the closest work in three specific ways. Guided Search Strategies in Non-Serializable Environments trains a 70B critic with temporal-difference learning on about 80,000 software trajectories. It uses the critic for one-step lookahead and trajectory selection, and doubles Qwen-72B's SWE-bench Verified success from roughly 20% to 40.8%. That critic assigns credit by regressing values along each trajectory separately; it never compares sibling runs of the same task. Our method supplies that method's inference-time mechanism, lookahead that scores actions before execution so no rollback is needed. We change how credit is assigned and move the setting to domain-knowledge tasks. DPRM derives implicit step rewards from final outcomes but needs a structured knowledge graph (Freebase), and AgentEval judges steps only after the fact and with a hand-built failure taxonomy. Neither exploits run-to-run variance as a label source, and neither steers a live agent.

Why This Regime

The domain setting is where the idea should matter most, and prior work constrains what the judge must look like. Brittleness and Promise trains medical judges over UMLS knowledge-graph paths. They score well at verifying paths but fail to improve downstream diagnosis on ProbSum. We read this as a warning against training a judge on a proxy structure that differs from the deployed task. Divergence labels are drawn from the deployed task itself, which is our proposed answer to that brittleness.

τ-Knowledge shows that even with gold documents placed in context, agents reach only about 40% on its τ-Banking domain. TheoremQA finds that injecting theorem definitions does not help. Together these rule out "just supply the knowledge" as a fix and point to errors in applying knowledge at particular hops. EHR-ChatQA reports that over half of consistent failures are value-linking errors, where the agent maps a colloquial term to the wrong database value. These are discrete, committed steps of the kind a fact-level divergence should catch.

Finally, The Silent Judge shows that prompted judges flip verdicts by up to 30% on provenance and recency cues and never acknowledge doing so. Tool outputs and documents in clinical and banking settings are full of such cues, so a prompted domain judge has a specific failure mode that a judge trained on sibling contrasts should partly avoid. Siblings share identical context, so surface cues cannot explain the label.

Research Program

The first component is a hop representation. We borrow the structured fact objects of REAP, whose fact extractor emits typed facts with a fulfillment status. After each agent turn, a fixed extractor records what the agent committed to: the entity or value it linked, the tool call and its arguments, the policy clause it invoked, and the claim it made to the user. Sibling runs are compared on these hop states rather than on raw tokens. Otherwise harmless wording differences would register as divergences.

The second component is divergence mining. For each training task we run the base agent $n$ times; we would use $n=8$ so that intermediate-$p_t$ tasks yield several successes and failures. For each failed run we align its hop sequence to the successful siblings. The positive label goes to the first hop whose committed content matches no successful sibling at the corresponding position; alignment is by tool type and slot, not step index. That hop becomes a negative example, and the matched hop from the closest successful sibling becomes a paired positive with the same prefix context. Hops before the divergence are labeled weakly positive. Hops after it are dropped, since AgentEval finds 63% of step failures propagate from upstream errors and later labels would be confounded.

The third component is the judge itself, a mid-sized open model fine-tuned pairwise on these contrasts to score a candidate hop given the trajectory prefix. The fourth is deployment. At each hop the agent proposes $K$ candidate actions, the judge scores them, and only the top one is executed. This keeps the method valid in stateful environments with simulated users, where whole trajectories cannot be replayed.

We would also measure how often the mined divergence hop matches the failure location a human annotator assigns, on a subset of about 150 failed EHR-ChatQA runs annotated with AgentEval's taxonomy. The method depends on this measurement: if divergence points are mostly noise, the rest of the program is moot.

Baselines

The naive reference is the unguided base agent on the same benchmarks, reported as Pass$^1$ through Pass$^k$. Published numbers anchor this reference. For o4-mini on EHR-ChatQA AdaptQA, the reported Pass@5 is 66.2% and Pass$^5$ is 7.5% (From Conversation to Query Execution). On τ-Banking, GPT-5.2 with terminal search reaches a 25.52% pass rate (τ-Knowledge); the file does not give its Pass$^4$ value. On the τ-bench airline domain, which serves as a general-domain control, claude-3-5-sonnet-20241022 falls from Pass$^1$ 0.460 to Pass$^4$ 0.225 (sierra-research/tau-bench). We rerun the base agents ourselves so that every comparison uses the same models and user simulators.

The strongest existing method is the temporal-difference critic of Guided Search Strategies, which reports 40.8% on SWE-bench Verified for guided Qwen-72B versus about 20% unguided. We found no reported result on Lacuna for that method on EHR-ChatQA, τ-Banking, or τ-bench. We reimplement it with λ = 0.7, the value that paper found best. It is trained on exactly the same rollouts our judge uses, with the same backbone, the same number of gradient steps, a matched hyperparameter search, and the same $K$ at inference. It is the fairest competitor because it uses the same raw ingredients, outcome-labeled trajectories and a learned step scorer, and differs only in credit assignment. Beating it means a statistically reliable Pass$^k$ improvement at matched $K$ and matched training rollouts on held-out tasks.

A second strong competitor is a prompted frontier judge used online. We would run GPT-4o with an AgentEval-style step rubric at the same $K$. AgentEval reports 0.89 offline failure-detection recall and κ = 0.84 with human experts on its own suites. The file gives no result for this judge in clinical or banking domains, and none for online use.

Two ablations isolate the key ingredient. The outcome-only ablation labels every hop of a failed run negative and every hop of a successful run positive, so the judge sees the same data without sibling alignment. The cross-task ablation pairs failed and successful runs from different tasks, which keeps the contrastive format but removes the shared-context control. If either ablation matches the full method, sibling contrast is not what is doing the work.

Why not just…?

Why not just use a stronger reasoning model? The reliability gap persists at the frontier. o4-mini's 66.2% versus 7.5% on EHR-ChatQA, o3-mini's drop from 37.93% to 14.57% precision between one and two hops on BioHopR, and GPT-5.2's 25.52% on τ-Banking all come from reasoning models. We still include the strongest available base model as an arm. If its Pass$^k$ alone exceeds a weaker agent guided by our judge, that is a reportable limit on the method.

Why not just sample K candidate actions and take a majority vote? This is the most serious cheap alternative, because voting directly reduces run-to-run variance and needs no training. Its weakness follows from the numbers. When Pass$^5$ is far below Pass@5, many tasks have $p_t$ well under one half, so the agent's most common action at the critical hop is often the wrong one, and voting makes the wrong choice more consistent. We include per-hop majority vote at matched $K$ and report results stratified by estimated $p_t$. If voting matches the judge on low-$p_t$ tasks, we would conclude the judge adds nothing and say so.

Why not just give the agent the right documents? τ-Knowledge's gold-document setting reaches only about 40%, and TheoremQA's theorem augmentation gives no significant gain. Knowledge presence is therefore not the bottleneck on these tasks. We run a gold-context arm on τ-Banking to confirm that the judge's gains add to, rather than duplicate, better retrieval.

Why not just prompt a frontier model as the step judge? It may work, which is why it is a main baseline and not a straw man. Two tests would separate it from our judge. The first is the head-to-head at matched $K$. The second is a cue-injection test adapted from The Silent Judge: authority and recency cues are inserted into tool outputs and policy snippets without changing their content, and we measure how often each judge's step choice flips. A trained judge that is less sensitive to such cues, at equal accuracy, would justify training even if headline Pass$^k$ ties.

Proposed Experiments

The main experiment uses EHR-ChatQA with a schema-transfer split. We mine divergences and train on MIMIC-IV tasks, then test on eICU tasks, so the judge must generalize to a renamed schema it never saw. Both IncreQA and AdaptQA are reported, with Pass$^1$, Pass$^5$, and the Pass@5 − Pass$^5$ gap. A second experiment uses τ-Banking with folds split by product category, reporting Pass$^1$ through Pass$^4$. τ-bench airline serves as a general-domain control, to test whether the method's benefit concentrates in domain settings as argued. All arms share the agent, user simulator, $K \in \{2, 4, 8\}$, and a matched token budget, and results come with bootstrap intervals over tasks.

Three checks decide the project. First, mined divergence hops must agree with human-annotated failure locations well above the rate of choosing a random hop before failure, on the 150-run annotated subset. If they do not, we stop and report the negative result. Second, at matched $K$, the sibling-contrast judge must beat both the temporal-difference critic and the outcome-only ablation on Pass$^k$ for held-out tasks. The advantage must be larger at $k = 4$ or $5$ than at $k = 1$, the signature of reduced variance rather than just a better best case. Third, stratifying test tasks by the base agent's estimated $p_t$, gains should concentrate in the intermediate band the equation above identifies. If gains instead appear only on tasks the agent almost always solves, the judge is polishing easy cases rather than closing the reliability gap.