When several language models are used as judges of the same answers, practitioners face two questions that current tools answer only by trial and error. First, how good could any way of combining these judges possibly be? Second, how much does each judge actually need to report, and which judges are redundant? This proposal answers both with one computable object. We treat the human quality label as a hidden signal and each LLM judge as a noisy sensor. We then compute, from a modest calibration set, the lowest prediction error that any combining rule could achieve when each judge's report is limited to a given number of bits. The result is a curve. Its right end is a hard ceiling on jury accuracy. Its shape shows how coarse judge outputs can be before information is lost. The gap between two versions of the curve, one where judges could coordinate and one where they report independently, measures what is lost because judges are queried separately.
The machinery comes from multi-terminal rate-distortion theory. This is the study of several separate encoders that each compress their own correlated observation and send it to one decoder. The version we use is the "CEO problem," where the decoder wants a hidden variable that none of the encoders sees directly. For our purposes the key fact is that under logarithmic loss (scoring a predicted probability distribution by the negative log-probability it gives the true outcome), these regions have exact single-letter characterizations for discrete alphabets (Courtade and Weissman). They can also be computed numerically with a generalized Blahut-Arimoto algorithm (Ugur et al.). The usual obstacle is that test-channel optimization is intractable in high dimensions. That obstacle mostly disappears here, because an LLM judge scoring on a five-point rubric emits a tiny alphabet.
Core Claim
Current multi-judge methods model judges as independent noisy readings of the true score. Instance-Optimal Estimation with Multiple LLM Judges on a Budget allocates queries and weights judges by inverse estimated variance and cost. LLM-as-a-Judge on a Budget does the same across items for a single judge. Neither can see that two judges who make the same mistakes carry little more information than one. Shared mistakes are the documented failure mode of LLM judges. Pairwise or Pointwise? shows that injecting assertiveness or prolixity flips pairwise preferences in about 35% of cases, and that assertiveness is especially effective on larger models.
Our claim is that a log-loss rate-distortion curve, estimated from judge outputs under a model with an explicit shared-bias variable, predicts three things: which judges add information, how many output levels each needs, and how close existing aggregators are to the best possible. It should do this better than variance-based reasoning, and the advantage should be largest under stylistic distractors, where shared bias grows.
Let $X$ be the human label, $Y_k$ the discretized score distribution reported by judge $k$, and $U_k$ a compressed message of at most $R_k$ bits derived from $Y_k$ alone. The quantity we compute is the distributed log-loss frontier
$$D_{\text{dist}}(R_1,\dots,R_K) = \min H(X \mid U_1,\dots,U_K),$$
where the minimum runs over per-judge compressions satisfying the Berger-Tung rate constraints. We compare it with the cooperative frontier
$$D_{\text{coop}}(R) = \min_{P(U \mid Y_1,\dots,Y_K):\ I(Y_1,\dots,Y_K;U)\le R} H(X \mid U).$$
The cooperative frontier is an information-bottleneck curve on the joint judge output, and it lower-bounds the distributed one at equal sum rate. Both curves approach $H(X \mid Y_1,\dots,Y_K)$ as rate grows. That limit is the log-loss floor no aggregator can beat.
Research Program
The first component is a model of the joint distribution of judges and labels that is sparse enough to estimate. With five judges, five-level outputs and a five-level label, the full joint table has about fifteen thousand cells. That is too many for HelpSteer2's roughly 21k responses to fill reliably. We therefore fit a latent-class model in which judges are conditionally independent given the pair $(X, B)$. Here $B$ is a small discrete shared-bias variable, for example "response is assertive or verbose in a way judges reward." We choose its cardinality by held-out likelihood. Setting $|B|=1$ recovers the conditional-independence assumption that implicit inverse-variance reasoning makes, which gives us a built-in ablation.
This construction raises a theoretical subtlety. The decoder wants $X$ only, while the remote source is $(X,B)$, and that "function of the remote source" case is not directly covered by the exact characterizations. We therefore report the Berger-Tung inner bound computed by the generalized Blahut-Arimoto iteration as the achievable curve. We report the cooperative bottleneck curve as the outer curve. Where the two meet, the answer is exact. Where they do not, we say so.
The second component turns the curves into decisions. The marginal information value of judge $k$ is the drop in the frontier when its rate goes from zero to its full output entropy. This is our proposed selection rule. The optimal test channel $P(U_k \mid Y_k)$ also says which score levels of judge $k$ can be merged without loss. That tells a practitioner, for example, that a cheap judge need only report a binary "contains a factual error" signal while a stronger judge should report the full five-point distribution.
The third component is a finite-sample analysis. The curves are estimates, so we bootstrap the calibration set. We report how many labeled items are needed before the estimated floor stabilizes. Asymptotic bounds are a known weakness when applied to finite data, and this analysis addresses it directly.
The role of prior work is as follows. The CEO and log-loss results supply the method (Courtade et al., Courtade and Weissman, Ugur, Aguerri, and Zaidi). The Gaussian vector results in the last of these do not transfer to discrete rubric scores, which is why we work with the discrete characterization. Collaborative Information Bottleneck motivates the cooperative-versus-distributed gap as the quantity of interest. It shows that this gap is zero for Gaussian sources but can be strictly positive for binary sources, and judge outputs are much closer to the latter. A Context-Aware CEO Problem constrains our expectations. It proves that decoder-side context stops helping once sensor noise dominates, so we do not claim gains from giving the aggregator references when judges are very noisy. DIBJudge applies an information bottleneck inside one judge's representations to remove translationese bias. We instead apply rate limits to the messages between judges and aggregator and do no fine-tuning. Distributional LLM-as-a-Judge motivates treating each judge's output as a distribution rather than a point, and trains single judges to do this. Its authors explicitly leave multi-judge aggregation open, and that is the gap this work fills.
Baselines
The naive reference is uniform static aggregation: every judge in the pool scores every item, and we take the mean score or majority vote. We convert the result into a predicted label distribution by the empirical histogram of votes. This is what most evaluation pipelines do. Any selection rule that cannot beat it at equal query cost has no practical value.
The strongest existing competitor on the judge-allocation question is Est-IVWE from Instance-Optimal Estimation with Multiple LLM Judges on a Budget. It explores judges, estimates per-judge variance, and allocates the budget in inverse proportion to variance and cost. It is the right opponent because it already optimizes a multi-judge budget with instance-optimality guarantees. Our only hypothesized advantage is that it ignores correlated errors. We run it on exactly the same judge pool, calibration split, test split and total query budget, with its exploration length tuned on the calibration split under the same tuning budget we give our method. The file reports no numerical result for Est-IVWE on HelpSteer2; no reported result found on Lacuna for Est-IVWE's error values. We will therefore reproduce it ourselves.
For the single-judge budget setting, the reference number is from LLM-as-a-Judge on a Budget. On HelpSteer2 correctness with GPT-4.1 nano at 50,000 queries, ROBIN-HOOD reaches worst-case absolute error 0.300 ± 0.008, against 0.451 ± 0.012 for uniform allocation. We run ROBIN-HOOD inside each selected jury, so our comparison isolates judge selection and quantization rather than per-item sampling.
For aggregation quality we include the Jury-on-Demand approach from Who Judges the Judge?. It trains XGBoost models on text features to predict per-instance judge reliability and weights judges accordingly, and it reports mean Kendall's Tau of 0.68 ± 0.02 on RAG groundedness. We reimplement it on our splits with the same features it describes. We also include a learned stacking aggregator: multinomial logistic regression on all judges' full score distributions. This is the empirical estimate of the infinite-rate floor.
Two ablations isolate the key ingredient. The first recomputes all curves and selection rules with $|B|=1$, that is, conditional independence. The second uses fixed uniform quantization of judge outputs in place of the optimized test channels.
Beating the baselines means the following. At matched query cost, bound-guided selection achieves lower held-out log-loss on human label distributions than Est-IVWE, with non-overlapping bootstrap intervals. The margin should widen on distractor-perturbed data, and the $|B|=1$ ablation should lose most of that margin.
Why not just…?
Why not just fit a stacking model and greedily add judges by validation log-loss? This is the most serious alternative. At infinite rate our floor equals what a perfectly fit stacking model would achieve, so for "which judges help" greedy stacking may do nearly as well when calibration data are plentiful. Our framework adds two things greedy stacking lacks: the finite-rate curve, which says how coarse each judge's report can be, and the cooperative-versus-distributed gap. We will test the threat directly. We run greedy stacking selection at calibration sizes from 200 to 5,000 items and compare it with bound-guided selection. If greedy stacking matches us at all sizes, we will report that the practical contribution reduces to output quantization and the aggregator-independent diagnostic.
Why not just estimate the full error covariance and use generalized least squares instead of inverse-variance weights? This fixes Est-IVWE's independence assumption at second order, and it may well capture much of the gain on mean-score estimation. It cannot capture shared biases that change the shape of the label distribution rather than its mean. It also says nothing about log-loss or about how many bits to request. We include a covariance-weighted variant of Est-IVWE as an extra baseline. If it closes the gap under distractors, our shared-bias latent is not buying anything beyond covariance.
Why not just use one strong distributional judge? Distributional LLM-as-a-Judge reports that distributional training cuts KL divergence to human label distributions on SNLI from 0.60 to 0.23 relative to single-point alignment. A single well-aligned judge may therefore beat a jury of unaligned ones. Our curves answer this quantitatively rather than by argument: if the floor for the jury is not below the single judge's held-out log-loss, the jury is not worth running. We report this comparison per dataset.
Proposed Experiments
We use HelpSteer2 (HelpSteer2) correctness and helpfulness labels. We also use SummEval (SummEval, code at Yale-LILY/SummEval), whose expert annotations give label distributions rather than single labels and whose Krippendorff's alpha of 0.71 for experts bounds how low log-loss can meaningfully go. The judge pool is five models spanning sizes, including Llama-3.1-8B and GPT-4.1 nano as in the budget papers. Each judge's output is read from rubric-token log-probabilities.
For the distractor condition, we follow the protocol of Pairwise or Pointwise?. We rewrite HelpSteer2 responses to be more assertive or verbose while holding correctness fixed. We then check the resulting flip rates against their reported 35% pairwise and 9% pointwise figures, to confirm the perturbation works.
Four decisive checks follow.
The first is calibration of the floor. The best learned aggregator's held-out log-loss should sit at or above the estimated floor $H(X \mid Y_1,\dots,Y_K)$ within bootstrap error. It should approach the floor as calibration data grow. An aggregator reliably below the floor would falsify the estimated model.
The second is prediction of marginal value. The rank correlation between each judge's predicted information gain and its realized log-loss improvement when added to the jury should be higher under the shared-bias model than under the $|B|=1$ ablation. It should also be higher than the ranking implied by inverse variance. The difference should grow under distractors, where our hypothesis says redundant judges share mistakes.
The third is rate. We will report the smallest per-judge bit budget at which the distributed curve comes within 0.02 nats of its infinite-rate value. We will then verify operationally that asking judges for correspondingly coarse, optimally merged outputs costs no measurable held-out accuracy. A useful finding would be that one to two bits per cheap judge suffice. A null finding, where full resolution is always needed, would itself tell practitioners that rubric granularity matters.
The fourth is the cooperation gap. We measure the distance between the distributed and cooperative curves at matched sum rate. Collaborative Information Bottleneck predicts this gap should be positive for discrete sources. If it is large, that argues for judge protocols in which judges see each other's verdicts. If it is near zero, independent querying is already sufficient. Either outcome is informative, and neither is known today.
References
- Multiterminal Source Coding Under Logarithmic Loss
- A generalization of blahut-arimoto algorithm to compute rate-distortion regions of multiterminal source coding under logarithmic loss
- Instance-Optimal Estimation with Multiple LLM Judges on a Budget
- LLM-as-a-Judge on a Budget
- Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
- Vector Gaussian CEO Problem Under Logarithmic Loss
- Collaborative Information Bottleneck
- A Context-Aware CEO Problem
- Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
- Distributional LLM-as-a-Judge
- Who Judges the Judge? LLM Jury-on-Demand: Building Trustworthy LLM Evaluation Systems
- HelpSteer 2: Open-source dataset for training top-performing reward models
- SummEval: Re-evaluating Summarization Evaluation