Many evaluations of AI output are now organized as trees of criteria. A broad goal such as "did the agent complete the shopping task" splits into sub-questions, and those split into concrete yes/no checks. Some checks are only meaningful if an earlier one passed: there is no point grading the price if the agent picked the wrong product. Running these trees with a large language model acting as judge (an "LLM judge") is accurate but slow, because each node usually gets its own call. Packing all nodes into one prompt is cheap, but judges then let unrelated criteria leak into one another's verdicts. The one method that fixes this leakage treats every interaction between criteria as a defect. That is wrong for trees, where a child's verdict is supposed to depend on its parent's.
We propose to train a small judge that scores an entire criteria tree in one pass. Its target for each node is a teacher's verdict on that node seen in isolation except for the verdicts of the node's ancestors. The student learns to use exactly the dependencies the tree permits and to ignore everything else. The final aggregation (gates, averages) stays in plain code. The model's only job is producing node verdicts that are both cheap to compute and free of contamination from non-ancestor criteria.
Core Claim
Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation shows that when several rubrics share one prompt, verdicts flip with rubric order, subset, or distractors. It traces this to attention leaking across rubrics. Its method, SARA, trains the joint-mode judge to match a moving-average copy of itself that sees each rubric alone. On ResearchQA this raises sample-level exact match for Qwen3-8B from 0.22 to 0.59. SARA's target is full isolation, and the authors concede that co-evaluating related rubrics might help. Hierarchical benchmarks make that concession concrete. Mind2Web 2 aggregates leaf checks with "gate-then-average" logic, where a failed foundational leaf fails its whole branch. FLASK groups 12 skills under 4 abilities. LLMCrit finds that clustering 3–5 related rules into aspects beats both monolithic prompts and single-rule prompts.
Our claim is that the right teacher for a tree is neither full isolation nor full joint context. It is ancestor-conditioned isolation. Let $T$ be a rubric tree, $x$ the response under evaluation, $r_v$ the rubric text at node $v$, $A(v)$ the ancestors of $v$, and $y_v$ its verdict. The student $p_S$ reads the whole tree and emits all verdicts in one decoding pass. We train it with
$$ \mathcal{L} = \sum_{v \in T} \mathrm{KL}\big(p_{\bar\theta}(y_v \mid x, r_v, y_{A(v)}) \,\|\, p_S(y_v \mid x, T, y_{<v})\big), $$
where $p_{\bar\theta}$ is the anchor model that sees only node $v$'s rubric plus the already-decided ancestor verdicts. The term $y_{<v}$ denotes whatever the student has generated before node $v$, including sibling verdicts. The loss pushes the student to make its verdict on $v$ invariant to siblings and to non-ancestor nodes, while staying sensitive to its ancestors. Setting $A(v)=\emptyset$ recovers SARA, so the comparison between the two is a single controlled ablation.
Research Program
The work has two stages, because the strongest tree judges are black-box APIs that expose only text.
In the first stage, a frontier teacher runs a tree-structured judge one node per call, with ancestor verdicts passed forward. This produces per-node rationales and verdicts. A compact open student, at the Qwen3-8B scale used by SARA and smaller, is fine-tuned on these as text targets, emitting the whole tree in one generation. This parallels how Prometheus distilled GPT-4 feedback into a 13B rubric judge, with one difference: Prometheus scores a single flat rubric per pass, and our targets are structured trees.
In the second stage, we apply the ancestor-anchored on-policy loss above. The anchor is a moving-average copy of the student, queried node by node, as in SARA. The only change from SARA is the conditioning set.
Aggregation from node verdicts to a task score is deterministic code that reproduces each benchmark's own rule. This keeps the learned part narrow and auditable, and it avoids asking a model to do arithmetic it does not need to learn.
We will also analyze whether attention mass from node $v$'s verdict tokens to non-ancestor rubric spans drops after training, while mass to ancestor spans is preserved. This mirrors SARA's attention diagnosis and tests whether the mechanism changes as claimed.
Baselines
The naive reference point is the base compact model prompted with the whole tree jointly and no training. SARA reports this condition at 0.22 exact match on ResearchQA for Qwen3-8B. It sets the floor for both accuracy and interference.
The cost-unconstrained reference is the teacher pipeline itself, meaning one call per node with ancestor context. It is the accuracy ceiling. On Mind2Web 2 the paper's Agent-as-a-Judge reports a 99.03% correctness rate against human experts. The student is not expected to beat this. Success means approaching it at a fraction of the calls.
The strongest existing method is SARA, which reports 0.59 exact match on ResearchQA for Qwen3-8B. We rerun it on identical data, the same student initialization, the same number of on-policy steps, and the same hyperparameter search budget. The only difference is its empty conditioning set. No reported result on Lacuna was found for SARA on Mind2Web 2 rubric trees or on FLASK's ability-skill hierarchy, so those numbers come from our reruns.
A flat distillation baseline in the style of Prometheus fine-tunes the student on the teacher's final aggregate scores and rationales, without per-node structure. Its reported human agreement is Pearson 0.897 on customized rubrics. It tests whether node-level supervision matters at all.
The key ablation trains a third variant whose anchor sees the full tree in isolation from nothing, which is plain joint self-consistency. It separates the effect of ancestor conditioning from the effect of on-policy self-training.
Beating SARA means three things together. The student must show equal or lower verdict flip rates under sibling permutation and distractor injection. It must show higher agreement with the per-node teacher specifically on gated child nodes. And it must hold up on flat ResearchQA, where the two methods coincide, as a regression check.
Why not just…?
Why not just use SARA's full isolation, since it already works? Full isolation is a real threat on flat checklists, and on ResearchQA we expect a tie. On trees it targets the wrong function. A child check such as "the reported price matches the source" has an ill-defined answer when the parent "identified the correct product" failed. A judge that never sees the parent verdict can only guess, so an isolation anchor will teach the student to guess. The experiment that settles this is the gated-node agreement metric under counterfactual parent flips, described below. If SARA matches our method there, the ancestor anchor adds nothing, and we will report that.
Why not just predict leaves independently and let code apply the gates? If every child verdict were well-defined without its parent, this would suffice, and it is cheap. We include it as a baseline: leaf-only training with SARA-style isolation plus coded aggregation. It fails exactly when a child's meaning shifts with its parent's outcome. Mind2Web 2's design of failing whole branches on foundational leaves suggests this happens often in agent tasks. We will measure how often it happens per benchmark. If the measured rate is negligible, the simpler design wins and the proposal narrows to interference removal only.
Why not just run a small model once per node? This avoids interference by construction. Its cost grows with tree size, which is the latency problem hierarchical judges already face. LLM-as-a-Judge on a Budget shows the scale of that cost: smart query allocation halves calls, from 100,000 to 50,000 for the same error. We report per-node small-model calls as a baseline in both accuracy and tokens. The claim is a better accuracy-per-token frontier, not higher raw accuracy.
Why not just prompt flat with chain-of-thought? Structured decomposition already beats flat prompting by large margins in the file's evidence. On ASAP, a Llama2-13b model using the multi-trait method from Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring reaches a quadratic weighted kappa of 0.560, against 0.407 for vanilla ChatGPT. Flat prompting is our naive baseline, not a serious competitor.
Proposed Experiments
The primary testbed is Mind2Web 2, whose 130 tasks come with native gate-then-average rubric trees. The student receives the agent's answer and the cached evidence that the extractor stage produced. We distill the verification tree, not browsing. Secondary testbeds are FLASK, with its two-level ability-skill hierarchy, and ASAP, using the four multi-trait dimensions from the essay-scoring paper as a shallow tree with ancestor-like dependencies between traits. ResearchQA and HealthBench serve as flat-rubric controls, where SARA and our method should coincide, and as a check on out-of-domain transfer.
The metrics are per-node agreement with the teacher and, where available, with human labels; task-level success agreement after coded aggregation; and the SARA-style flip rate under sibling reordering, subset removal, and distractor rubrics. Benchmark-native scores are also reported: quadratic weighted kappa on ASAP and correlation with human scores on FLASK. Cost is reported as calls and tokens per evaluated response.
The decisive check is counterfactual. For a sample of Mind2Web 2 and ASAP items, we minimally edit the response so that one parent node's verdict flips while leaving the child's evidence untouched. A good tree judge changes the child verdict exactly when the teacher pipeline does, and is unchanged by edits to siblings.
The predicted signature of our method is low sibling sensitivity, matching SARA, combined with high ancestor sensitivity, where SARA is near zero. A null result, with SARA indistinguishable from ancestor anchoring on this test, would show that tree dependencies can be safely ignored at the node level. That would argue for the simpler leaf-plus-code design.
A secondary check reuses the provenance and recency cue injections from The Silent Judge, which reports a +30% verdict shift for GPT-4o under recency cues on ELI5. This tests whether node-level grounding reduces shortcut sensitivity relative to the flat-distilled student. We make no prior claim here.
References
- Mitigating Rubric Interference in LLM Judges via On-Policy Self-Distillation
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
- FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
- LLMCrit: Teaching Large Language Models to Use Criteria
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models
- LLM-as-a-Judge on a Budget
- Unleashing Large Language Models' Proficiency in Zero-shot Essay Scoring
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge