Automated evaluation of language models increasingly runs as a small "editorial board" of models. A cheap verifier screens outputs, specialist agents each hunt for one kind of error, and a manager agent arbitrates. Sitting above all of them is a scarce human gold label. These boards agree with people better than a single judge does, but they are expensive. The usual way to cut the cost is to send only the hard cases upward, which manufacturers would call pulling expensive work only when demand calls for it. Doing that quietly breaks the statistics: once a model-chosen subset decides which items reach the human tier, a simple average of the final scores is biased and its error bars are wrong.
This note proposes making escalation between tiers randomized with probabilities the system records. Those probabilities are set by how much downstream users care about an item (its "demand") and by how uncertain the lower tiers are about it. A chained correction estimator then turns the cheap scores, the deep scores and the few human labels into a single quality estimate with valid confidence intervals, whatever the quality of the cheap tiers. The practical outcome is a review board that spends its expensive tiers where the uncertainty and the stakes are concentrated, and that reports a number a reader can trust.
Core Claim
Multi-tier judging and statistically valid judging have so far been built separately. HiMATE builds a two-tier specialist-and-manager agent board for machine translation error detection. It sends every segment through the full board, at roughly 1.5 to 2 times the tokens of simpler judges, and outputs a heuristic weighted score. Multi-Agent-as-Judge (MAJ-EVAL) uses persona panels with debate and an aggregator, at about $0.42 per task, and likewise reviews every input uniformly. On the statistics side, Efficient Inference for Noisy LLM-as-a-Judge Evaluation derives estimators that combine a fixed human calibration set with many noisy judge scores and yield valid intervals. Its setup is flat: one surrogate and one gold tier, with the calibration set fixed in advance.
The claim is that a review board becomes statistically valid once it is treated as a multi-phase sampling design. In that design each tier is a surrogate for the tier above it, and the decision to go up a tier is a randomized draw with a logged probability. This buys two things. The estimate stays unbiased even when the cheap tiers are poor or miscalibrated. And escalation probabilities can be chosen in advance to minimize interval width per unit of cost, weighted by downstream demand.
The closest dynamic-evaluation work is Chatbot Arena. It uses active sampling to choose which model pairs users compare and reports needing 54% less data to reach a rating precision of 0.2. It allocates comparisons rather than review depth, and it has no tiers. This proposal differs from all three lines in one specific way. Routing to deeper review is never deterministic, so the routing itself can be corrected for in the estimate.
Research Program
The board is built from modules that share one record format, in the style of OmniEvaluator and the Universal Causal Evaluation Engine. OmniEvaluator splits evaluation into data, inference, post-processing and metric stages linked by a shared intermediate schema. The causal engine standardizes fit and predict calls so estimators can be swapped. We reuse that design discipline here, with one addition: every module writes into the record the probability with which an item reached it. The modular design is therefore not decoration. It is what makes the tiers interchangeable for ablations, and it is what makes the logged probabilities trustworthy.
There are four tiers. Tier 0 is a cheap verifier, such as OmniEvaluator's 0.6B model or a reward model. Tier 1 is a set of specialist agents. Tier 2 is a manager agent that arbitrates among the specialists. Tier 3 is the human label.
The quantity we estimate is the demand-weighted mean quality $\theta = \sum_i w_i Y_i / \sum_i w_i$. Here $Y_i$ is the human-tier score of item $i$, and $w_i$ is its downstream demand weight, for example how often that type of query occurs in real use. For each item, the indicator $R_{ik}$ equals 1 if the item was escalated to tier $k$, which happens with known probability $\pi_{ik}$. That probability may depend on everything observed at lower tiers. Each $f_k$ is any predictor of $Y$ built from the outputs of tiers $0$ through $k$, fitted by cross-fitting (fitting on held-out folds so the predictions are not tuned to the items they score). The chained estimator is
$$\hat\theta = \frac{1}{\sum_i w_i}\sum_i w_i\Big[f_0 + \tfrac{R_{i1}}{\pi_{i1}}(f_1 - f_0) + \tfrac{R_{i1}R_{i2}}{\pi_{i1}\pi_{i2}}(f_2 - f_1) + \tfrac{R_{i1}R_{i2}R_{i3}}{\pi_{i1}\pi_{i2}\pi_{i3}}(Y_i - f_2)\Big].$$
Every correction term has mean zero because the probabilities $\pi$ are known. The estimator is therefore unbiased for any choice of $f_k$, and good choices of $f_k$ only shrink the variance. This mirrors how the flat estimator in the noisy-judge paper behaves, extended here to a chain of tiers.
The allocation rule follows classical two-phase sampling. Escalation to tier $k$ is set roughly proportional to $w_i \hat\sigma_{ik} / \sqrt{c_k}$, where $\hat\sigma_{ik}$ is the predicted residual spread of the next tier given the current one and $c_k$ is the cost of that tier. A floor on each $\pi$ keeps the weights bounded. The result is that high-demand items on which the lower tiers are unsure get pulled upward, while low-stakes items on which the tiers agree are settled cheaply.
Two smaller analyses ride on the same records. The first adapts ordinal-versus-cardinal score calibration, of the kind used in the NIPS 2016 review audit, to detect score inflation in agent tiers. That audit found that nearly 60% of submissions received the score intended for the top 30%. The second asks whether escalation concentrates on a measurable "messy middle" of uncertain items, analogous to the band of papers whose fate the audit found close to random.
Baselines
The naive reference point is a single Tier 0 judge whose raw average is reported with a textbook interval. It shows what is lost by having no calibration at all.
The flat statistical baseline is the efficient-influence-function estimator and PPI++ from Efficient Inference for Noisy LLM-as-a-Judge Evaluation. It uses one judge plus a uniformly sampled human calibration set. That paper reports intervals 35–55% narrower than standard prediction-powered inference and 3–15 times narrower than the Rogan-Gladen correction. It is the strongest competitor on the validity question. It receives the same total budget in dollars and tokens and the same human-label budget, and its single judge may be any tier, including the most expensive.
The strongest agreement-oriented baselines are HiMATE run on every item and MAJ-EVAL run on every item. The full board's score is then calibrated with the same flat estimator, so neither is penalized for missing statistics. For HiMATE the file reports only relative gains, a 95% improvement in recall and an 89% average improvement in F1 for error span detection on MQM22 and MQM24, so no absolute F1 is available to quote. On StorySparkQA the file reports Spearman correlations with human judgment of 0.47 for MAJ-EVAL, 0.31 for G-Eval and 0.15 for ROUGE-L, and 0.40 for MAJ-EVAL on MSLR-COCHRANE.
The key ablations isolate the proposal's ingredients. The first is a deterministic confidence-threshold cascade: escalate if the cheap tier's confidence falls below a cutoff, then report the top tier's average. The second keeps the randomized escalation but with uniform probabilities, removing the demand-and-uncertainty allocation. The third sets every $w_i$ to the same value, removing the demand weighting.
All methods share data splits, model backbones, the human-label budget and the hyperparameter tuning budget, and every tuning step is done on a separate set of items. Beating the baselines means two things at the same cost. First, nominal 95% intervals must actually cover the true value about 95% of the time. Second, the intervals must be narrower than those of the flat estimator and of the calibrated full board, by a margin larger than the variation across random seeds.
Why not just…?
Why not just escalate when the cheap judge is unsure, like any model cascade? This is the main threat, and it is included as an ablation rather than argued away. A deterministic cascade measures humans and deep tiers only on items the cheap tier flagged. Everything below the cutoff is scored by a model that is never checked, so the overall estimate inherits that model's bias in exactly the region nobody looks at. If the experiments show that the cascade's bias is negligible on MQM and Arena data, and that its coverage is near 95%, then the randomization adds little. In that case the contribution narrows to the allocation rule plus a guarantee for cases where the cheap tier is poor, and we would report it that way.
Why not just use one judge with the efficient-influence-function correction? The flat estimator is already valid, so tiers can only help by shrinking variance. They earn their cost only if each deeper tier explains residual human variance that the tier below cannot. We measure that variance reduction per unit cost directly. If the specialist and manager tiers add little beyond the best single judge, the flat baseline wins and the proposal fails on that dataset. MAJ-EVAL's gain over G-Eval suggests the deeper tiers do carry extra signal, but a gain in rank correlation is not the same as a gain in variance reduction.
Why not just run the full board on everything and calibrate afterwards? This is valid, but it spends 1.5 to 2 times the tokens of a simple judge, by HiMATE's own account, on items where the tiers already agree. The matched-budget comparison settles whether moving that spend into more human labels, or into deep review of high-demand items, yields narrower intervals.
Why not just escalate items uniformly at random? This is the uniform-probability ablation. Classical two-phase sampling theory predicts it will be beaten when residual uncertainty or demand is uneven across items. Chatbot Arena's finding that uncertainty-driven allocation needs 54% less data is suggestive here, but it concerns choosing comparisons rather than choosing review depth.
Proposed Experiments
Every experiment is run as a semi-synthetic replay. MQM22 and MQM24 have human error annotations for every segment. The human tier is "revealed" only for the items the policy escalates, and the true value of the estimand is computed from all the labels. This is the same proxy-truth logic the causal evaluation engine uses for counterfactuals that cannot be observed.
On MQM, the demand weights are domain and segment-length strata with stated, varied weightings. We are explicit that this demand is simulated. On Chatbot Arena's arena-human-preference-140k data, demand weights come from topic-cluster frequencies. Chatbot Arena reports about 600 clusters in real use, with the largest holding only 1% of queries. On StorySparkQA and MSLR-COCHRANE, every item has an expert rating, so the same replay applies, with Spearman agreement as a secondary metric.
The decisive check is coverage. Across 1,000 replays at each budget, does the chained estimator hold 95% coverage while the threshold cascade and the raw board score fall short? The second check is the width of the confidence interval as a function of total cost, plotted against the flat estimator and the calibrated full board. The third is a stress test in which a deliberately miscalibrated or score-inflated Tier 0 is swapped in. The chained estimator should remain unbiased while the cascade should drift.
Two further analyses complete the study. One measures the fraction of human variance explained at each tier, which shows where the board's money buys information. The other is the inflation audit adapted from the NIPS 2016 review study. If coverage holds and intervals narrow at matched cost on at least two of the three data families, the claim stands.
References
- HiMATE: A Hierarchical Multi-Agent Framework for Machine Translation Evaluation
- Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation
- Efficient Inference for Noisy LLM-as-a-Judge Evaluation
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- A Composable Evaluation System for Reproducible Omni-Modal Foundation Model Evaluation
- Universal Causal Evaluation Engine: An API for empirically evaluating causal inference models
- Design and Analysis of the NIPS 2016 Review Process