Agent systems built from language models are increasingly graded step by step. Another model, an "LLM judge," reads each planning step, tool call, or turn and decides whether it was correct. When steps depend on one another, the judge's verdicts are chained too. Deciding that step 5 failed "because of" step 2 only makes sense if the judge got step 2 right. We already know that agents snowball their own mistakes. A wrong early turn drags later turns down because models favor consistency with their own history over accuracy. Nobody has asked whether the judges grading those agents snowball in the same way. This proposal measures exactly that. We test whether one wrong or prompt-sensitive verdict early in a trace makes the judge's later verdicts and its root-cause diagnosis wrong more often than chance. We then test whether propagating each verdict's measured instability through the dependency structure contains the damage.
The question matters because step-level judging is being adopted precisely to diagnose cascades. AgentEval models agent workflows as directed acyclic graphs (DAGs: dependency diagrams with no loops). It grades each node with a calibrated GPT-4o judge and reports that 63% of step failures propagate from upstream. Its attribution logic treats each upstream verdict as ground truth. If the judge is itself a cascading system, the tool built to find snowballs becomes a new source of them, and its root-cause accuracy is limited by the judge's own fragility rather than by the agent's.
Core Claim
We claim that step-level judges spread their own errors through two distinct channels. Only one of these is fixed by the obvious remedy. The first channel runs through context. When the judge sees its own earlier verdicts or rationales, it anchors on them. This mirrors the "consistency over accuracy" behavior that the multimodal snowballing study documents in generators. The second channel is structural and survives even if every step is judged in a fresh context. DAG attribution rules use upstream verdicts to decide downstream blame. A false "fail" upstream wrongly excuses the downstream step, and a false "pass" upstream wrongly blames it. Because of the structural channel, we predict that small, meaning-preserving rewordings of the judge's rubric will flip trace-level root-cause diagnoses more often than they flip any individual step verdict. The trace-level diagnosis amplifies step-level brittleness.
Our second claim concerns the remedy. A judge's own prompt sensitivity can be turned into a per-step reliability estimate. We do this by measuring how often a step's verdict flips across rubric paraphrases, in the spirit of SENSEL, which uses instability under perturbation to predict errors in single-turn classification. We then propagate that estimate through the DAG, so that attribution is averaged over plausible upstream verdicts instead of resting on single point decisions. We predict this beats both hard-verdict DAG attribution and the strong positional heuristic on root-cause accuracy, and that it makes diagnoses stable under rubric rewording.
Relation to Prior Work
Two papers frame the gap. Flaw or Artifact? replaces string matching with a Gemini judge and finds that apparent student prompt sensitivity largely disappears. On ARC-Challenge, for example, the standard deviation across 12 prompt templates falls from 0.28 to 0.005. That study treats the judge as invariant and evaluates only single-turn answers. RULERS shows the judge is not invariant: rubric reordering and paraphrase make scores drift. It counters this with locked rubric specifications, verbatim evidence quotes, and distribution calibration, but only for single essays and summaries. Both papers motivate us. Neither studies a judge whose verdicts feed one another.
AgentEval supplies the setting and the strongest judge-based method. We reuse its DAG attribution, and we relax its assumption that upstream verdicts are correct. AgentTrace constrains us. Its causal tracing reaches 94.9% Hit@1 on injected failures, and position alone gets 87.3%, against 68.5% for a GPT-4 judge. Any judge-based attribution must therefore clear a very simple heuristic. Brittlebench supplies the analysis method. Its variance decomposition, which separates item difficulty from prompt sensitivity, is what we extend from single items to whole traces. PPTC and MTU-Bench supply something unusual: deterministic per-turn checkers. These let us score the judge against ground truth without human labels. MTU-Bench's authors reject LLM judges as biased and irreproducible. We take that critique seriously by working only in domains where we can check the judge.
Research Program
The first stage builds traces with known step-level truth. We run several agents of different strength on PPTC sessions (2–17 turns) and MTU-Bench multi-turn multi-tool dialogues. We label every turn correct or incorrect with PPTX-Match and with MTU-Bench's turn-level checks, using the MTU-Bench code. The first failing turn on a failing branch serves as the ground-truth root cause. Following AgentTrace's protocol, we also inject single controlled faults into otherwise correct traces, such as a wrong slide index or a wrong tool argument. This gives cases where the root cause is known exactly and is not always at the first position. That matters, because otherwise position trivially wins.
The second stage measures judge cascades. For step-level judges (GPT-4o-class and an open-weight judge), we define the amplification of judge error as
$$A = P(e_{t+1} \mid e_t) - P(e_{t+1} \mid \neg e_t),$$
where $e_t$ means the judge's verdict on step $t$ disagrees with the checker. We estimate $A$ separately for steps where the agent was right and where it was wrong, so that a real agent cascade is not mistaken for a judge cascade.
To isolate the context channel causally, we run a counterfactual. We edit the judge's visible prior verdict on step $t$, flipping "pass" to "fail" while leaving the agent's actions untouched, and measure how often the verdict on unchanged downstream steps moves. To isolate the structural channel, we judge every step in a fresh context and count how many root-cause errors come purely from the attribution rule consuming a wrong upstream verdict.
For prompt sensitivity, we generate rubric rewordings under Brittlebench's perturbation taxonomy and compare two quantities. One is the step-verdict flip rate. The other is the root-cause flip rate per trace, plotted against trace depth.
The third stage is the remedy. For each step $i$, we compute $s_i$, the fraction of $K$ rubric paraphrases (we plan $K = 8$) under which the verdict disagrees with the majority. We map $s_i$ to a probability $q_i$ that step $i$ truly failed, calibrated on a held-out split. The root cause is then the step with the highest probability of being the earliest true failure on the failing path. Assuming independence, the probability that step $j$ is the earliest true failure is
$$P(\text{root} = j) = q_j \prod_{i \in \text{anc}(j)} (1 - q_i),$$
where $\text{anc}(j)$ is the set of steps upstream of $j$. Hard-verdict AgentEval attribution is the special case where every $q_i$ is 0 or 1. We also test a variant that multiplies in a positional prior, because AgentTrace shows position is informative.
Baselines
The naive reference points are two. The first is end-to-end outcome checking, which in AgentEval detects failures with recall 0.41. The second is the positional heuristic "blame the earliest step on the failing branch." AgentTrace reports 87.3% Hit@1 for this heuristic on its synthetic suite, and we rerun it on our traces.
The strongest existing judge-based method is AgentEval's DAG-structured step judging with hard-verdict attribution. That paper reports 72% root-cause accuracy against an 81% human ceiling, failure detection recall of 0.89, and Cohen's κ = 0.84 with experts, all on its own production data. No reported result was found on Lacuna for AgentEval-style judging on PPTC or MTU-Bench, so we rerun it ourselves on identical traces with the same judge model. We give it the same total judge-call budget as our method: $K$ calls per step, used for majority voting over resampled or reworded prompts.
The strongest robustness method on the judge side is RULERS-style locked rubrics with evidence gating. We apply its rubric locking and verbatim-evidence requirement to step judging. RULERS reports QWK 0.7276 versus 0.4319 for holistic scoring on ASAP 2.0, but has no reported result on agent traces.
Two ablations isolate the key ingredient. In the first, $q_i$ comes from temperature resampling of a single fixed rubric instead of rubric paraphrases. This tests whether prompt variation carries information beyond ordinary sampling noise. In the second, $q_i$ comes from the judge's verbalized confidence or verdict-token probability, matching SENSEL's MaxProb and entropy comparison. SENSEL reports +4.1 AUC over those signals on GPT-J-6B.
Our method counts as beating the baselines if it does two things. It must exceed both hard-verdict DAG attribution and the positional heuristic on root-cause Hit@1, on injected faults placed away from the first step, by a margin larger than the bootstrap interval. And it must reduce root-cause flip rate under rubric rewording at matched call budget.
Why not just…?
Why not just hide the judge's earlier verdicts and grade every step in a fresh context? This removes the context channel, and we expect it to help, so it is a real baseline rather than a straw man. It cannot remove the structural channel, because DAG attribution must read upstream verdicts in order to assign blame. AgentEval shows that flat grading without dependency edges loses recall and root-cause accuracy. Our fresh-context condition measures exactly how much root-cause error remains once the context channel is gone. If that remainder is negligible, the second half of this proposal is unnecessary, and we will report that.
Why not just use deterministic checkers? Where they exist, they are better, and we use them as ground truth for that reason. Most agent steps, though, are free-form plans or tool choices with no checker. AgentEval reports that outcome checks miss most intermediate failures (recall 0.41). The point of this study is to learn how much to trust judges in the domains where checkers are missing, by calibrating them in domains where checkers exist. A transfer test, described below, checks whether that calibration carries over.
Why not just blame the first step? AgentTrace makes this the most dangerous alternative: at 87.3% Hit@1 it beats an LLM judge by almost 19 points. If position wins on our natural traces as well, step-level judging is not worth its cost for root-cause localization. We therefore report natural traces and off-first-position injected faults separately. We also test whether combining position with $q_i$ beats position alone.
Why not just resample at nonzero temperature or lock the rubric? Resampling is cheaper than writing paraphrases, and locked rubrics might remove the brittleness that our signal depends on. The first ablation settles the resampling question. For RULERS-style locking, we predict it reduces flip rates but not systematic upstream misjudgments. That would leave correlated errors that even a stable judge still cascades. We will run locked rubrics both as a competitor and as the base judge underneath our attribution.
Proposed Experiments
The first decisive check tests whether judge cascades exist. We expect $A$ to be significantly positive when prior verdicts are visible, and the counterfactual verdict flip to move downstream verdicts on unchanged steps. A null result here would falsify the context-channel claim.
The second check tests amplification. Root-cause flip rate under rubric rewording should exceed the per-step flip rate and grow with trace depth on PPTC. This is the first evidence that judge brittleness compounds, and it directly qualifies the invariance assumed by Flaw or Artifact?.
The third check is the attribution comparison described under Baselines. It is run on the PPTC and MTU-Bench natural and injected-fault splits, with root-cause Hit@1 as the primary metric and selective-prediction AUC for step verdicts as the secondary metric.
The fourth check tests transfer. We fit the $s_i \to q_i$ calibration on PPTC and apply it unchanged to MTU-Bench, to see whether perturbation-based reliability carries across tool domains. We also report Brittlebench-style variance decomposition at the trace level, separating trace difficulty from judge-prompt variance. The result would tell practitioners how much of a step-level judge's diagnosis reflects the agent and how much reflects the wording of the judge's own instructions.
References
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Flaw or Artifact? Rethinking Prompt Sensitivity in Evaluating LLMs
- RULERS: Locked Rubrics and Evidence-Anchored Scoring for Robust LLM Evaluation
- AgentTrace: Causal Graph Tracing for Root Cause Analysis in Deployed Multi-Agent Systems
- Brittlebench: Quantifying LLM robustness via prompt sensitivity
- PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task Completion
- MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models