When several large language models act as judges, for example voting on whether an answer is relevant or a comment is toxic, their votes are not independent. Models trained on overlapping web data and similar alignment targets tend to make the same mistakes. A recent method fixes the vote-counting side of this problem. It models the panel's joint votes as an Ising model, a probabilistic model of binary variables with pairwise "coupling" weights, so that it can learn which judges err together. It then infers the hidden true label without any ground truth, using expectation-maximization (EM). EM is an algorithm that alternates between guessing the hidden labels and refitting the model. Unsupervised EM is known to be fragile: it can settle into a wrong but self-consistent explanation of the data. For judge panels there is a specific and dangerous wrong explanation. A large group of correlated judges agrees on an error, and the fitted model treats that group's shared error as the truth.
This note proposes to fit dependence-aware aggregators with deterministic annealing. The hidden-label guesses start fully uncertain and are allowed to sharpen only gradually, so the fit follows a single smooth path instead of committing early to whatever the loudest group of judges says. The temperature path also does a second job. The point along the path at which a judge's couplings "freeze" gives a label-free measure of how redundant that judge is, which can be used to shrink the panel. Neither use has been tested. The note is written so that a negative answer, where cheaper fixes like smart initialization or random restarts work just as well, would still settle the question cleanly.
Core Claim
The claim has two parts. First, in fully unsupervised aggregation with correlated subgroups of judges, standard EM for the class-dependent Ising model converges to the "clique-as-truth" solution at a measurable rate. Annealed EM reduces that rate below what equal-compute random restarts or initialization from a simpler independent-judge model achieve. Second, the inverse temperature at which each judge's couplings stabilize ranks judges by redundancy. Pruning by that ranking keeps aggregation accuracy with fewer judges than pruning by raw pairwise agreement. Raw agreement cannot separate "agree because both are right" from "agree because both share a blind spot."
Background and Positioning
The aggregation model comes from Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models, with details in its companion version. That work models the vote vector $v \in \{-1,1\}^K$ of $K$ judges, conditional on the hidden label $y$, as
$$p(v \mid y) \propto \exp\big(h_y^\top v + \tfrac{1}{2} v^\top W_y v\big),$$
where $h_y$ holds each judge's per-class bias and $W_y$ holds class-specific judge-judge couplings. It fits these parameters with generalized EM on a pseudo-likelihood and proves that ignoring couplings leaves excess risk that does not vanish as the panel grows. This paper motivates the proposal and supplies the model. It fits at a single fixed temperature and does not study which solution EM lands in. We keep its model unchanged and change only how it is fitted and what is read off the fit.
The fitting technique comes from Deterministic Annealing Variant of the EM Algorithm. That paper replaces EM's posterior over hidden variables with a tempered one:
$$q_\beta(y \mid v) \propto p(v, y; \theta)^\beta, \qquad \beta: 0 \to 1,$$
where $\beta$ is the inverse temperature. A small $\beta$ gives a nearly uniform posterior and a smooth objective with one basin. The fit is then tracked as $\beta$ rises to 1. That paper demonstrated the idea only on a one-dimensional two-component Gaussian mixture. It also warns that tracking can stall at saddle points and bifurcations. That warning constrains us: the first symmetry break along the path may follow the dominant correlation direction in the votes, which could be the clique itself. So annealing is not guaranteed to help, and we treat that possibility as a hypothesis to test rather than a detail.
Two other lines of annealing work show that temperature continuation fixes premature commitment in structured problems. Deterministic Annealing for Semi-Supervised Structured Output Learning anneals relaxed label distributions in conditional random fields when labels are scarce. ASAP: Architecture Search, Anneal and Prune anneals a softmax over candidate operations and prunes those that fall below a threshold. ASAP's pruning-during-annealing pattern is the template for our judge-pruning diagnostic. The difference is that we prune judges by when their couplings stabilize along the temperature path, not by mixture weight.
The broader ensembling evidence explains why this matters. Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification shows that disagreement across model families catches confident errors. Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap shows that pools of modern models can be nearly redundant (pairwise Q-statistic 0.961). It also shows that aggressive combiners over-sharpen probabilities. That second finding is why we report the calibration of the aggregated posterior and not only accuracy.
Research Program
We first build a controlled testbed of synthetic panels with known truth. In these panels, $K$ judges have varied accuracies, and a clique of $m$ judges shares an error pattern on a fraction $\rho$ of items. We sweep the clique size relative to the panel, from a minority up to a majority, and sweep $\rho$. For each setting we map the solutions EM can reach. We record how often each fitting procedure lands in the truth-aligned solution versus the clique-aligned one, and whether the clique-aligned solution has higher pseudo-likelihood than the true one. That last question matters. If the wrong solution is the better-scoring one, no optimizer can fix the problem, and the result shows an identifiability limit rather than an optimization one.
Next we implement annealed EM for the class-dependent Ising model in two variants. The first anneals only the label posterior, as in the deterministic annealing EM paper. The second also delays the couplings: $W_y$ is scaled by $\beta$, so the model passes through an independent-judge regime before couplings can absorb correlations. This variant tests a specific mechanism, namely that couplings learned before the label posterior has sharpened are what let a clique "explain itself."
We also replace geometric cooling with an adaptive schedule. The schedule takes smaller temperature steps when the posterior entropy changes fast, and it checks for bifurcations by looking at the leading eigenvalue of the pseudo-likelihood Hessian. That check responds directly to the saddle-point limitation noted in the deterministic annealing EM paper.
Finally, we record each judge's couplings along the path and define its freezing temperature: the $\beta$ beyond which its row of $W_y$ changes less than a tolerance. Judges whose couplings freeze early and strongly are candidates for redundancy. We test whether removing them, refitting, and comparing accuracy produces a better accuracy-versus-panel-size curve than the alternatives.
Baselines
The naive reference is unweighted majority vote, together with the conditional-independence weighted vote (CI-WMV) from the Ising paper. That paper reports CI-WMV accuracy of 0.820 on WikiQA relevance, 0.694 on Jigsaw toxicity, and 0.737 on CNN/DailyMail summarization judgments. These numbers show how much dependence modeling already buys.
The strongest existing method is the class-dependent Ising model fitted by generalized EM, as published. It reports accuracy of 0.912 on WikiQA, 0.792 on Jigsaw, and 0.801 on CNN/DailyMail (source). Our judge panel will differ from theirs, so these numbers cannot be compared directly. We will reimplement their fitting and run it on our exact vote matrices. We will first check that, on a panel as close as possible to theirs, we come within a reasonable margin of their reported figures.
Two stronger optimization baselines are run under the same wall-clock budget as annealing:
- Random restarts. Ising EM with $R$ random restarts, keeping the restart with the highest pseudo-likelihood. $R$ is set so that total compute matches the annealing run.
- Dawid-Skene initialization. Ising EM initialized from a converged Dawid-Skene fit, a classical aggregation model that assumes judges are independent given the label. This is a one-step continuation and the most likely real-world default.
All methods share the same stopping tolerance and the same small hyperparameter search over regularization strength. Hyperparameters are selected by unsupervised pseudo-likelihood, never by labels.
We also run ablations that isolate the key ingredient: posterior annealing alone, coupling delay alone, both together, and fixed geometric versus adaptive schedules. For pruning, the baselines are removing judges by highest raw pairwise agreement, removing judges by lowest estimated accuracy, and greedy backward elimination by pseudo-likelihood.
Beating the baselines means a statistically significant reduction in the clique-capture rate on synthetic panels where the truth solution is the likelihood maximum. On the planted-clique real panels, it means higher accuracy and lower aggregated-posterior calibration error than both the equal-compute restart baseline and the Dawid-Skene-initialized baseline. Matching them on clean panels is acceptable. Losing to them anywhere is a reportable negative result.
Why not just…?
Why not just run EM from many random starts and keep the best? This is a real threat. If the truth-aligned solution is the pseudo-likelihood maximum, enough restarts will find it, and annealing's only advantage would be determinism and compute. The synthetic sweep settles this directly. We measure, at matched compute, how many restarts are needed to match annealing's success rate. We also measure how often likelihood-based selection among restarts picks the wrong solution because the two solutions score similarly. If restarts match annealing at equal compute, we will say so, and the contribution narrows to the solution-structure analysis and the pruning diagnostic.
Why not just initialize the Ising fit from Dawid-Skene or majority vote? This is the most likely reason annealing turns out unnecessary, because a sensible initialization already sits near the truth when most judges are independent and accurate. The case where it should fail is exactly the planted-clique case. There, the independent-judge fit itself is captured by the clique, because it counts the clique's correlated votes as independent evidence. The coupling-delay variant is designed so that it can beat this baseline only if the order in which labels sharpen and couplings switch on matters. Comparing coupling-delay against Dawid-Skene initialization is therefore the decisive test of whether the whole proposal is needed.
Why not just label a few dozen items and fit semi-supervised? In many deployments this is the cheapest fix, and we do not argue against it. We will measure how many gold labels are needed for supervised anchoring to remove clique capture. Annealing is useful only in the regime below that count, such as new tasks, private data, or continuously changing judge pools. That count is a useful quantity for practitioners whichever side of it annealing ends up on.
Why not just drop judges that agree too much? Raw agreement conflates shared correctness with shared error. Two excellent judges agree almost always. The comparison against agreement-based pruning tests whether freezing temperatures, which are computed with the label posterior in the loop, actually separate the two.
Proposed Experiments
The synthetic study runs panels of 5 to 20 judges with planted cliques across the clique-size and error-rate grid. It reports clique-capture rate, aggregation accuracy, calibration error of the aggregated posterior, and wall-clock time. Each of these is reported separately for settings where the truth solution is and is not the pseudo-likelihood maximum.
On real data we use the three tasks from the Ising paper: WikiQA (WikiQA: A Challenge Dataset for Open-Domain Question Answering), Jigsaw toxicity (Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification), and CNN/DailyMail (Teaching Machines to Read and Comprehend). We collect votes from a panel of open-weight families like the jury in the cross-model disagreement paper: Llama, Mistral, Qwen, Gemma and Granite. We then deliberately add several fine-tunes or prompt variants of one base model. This plants a genuine correlated clique, which we grow from minority to majority.
Three outcomes would settle the project:
- Clique-capture test. Whether coupling-delayed annealing keeps accuracy flat as the planted clique grows while Dawid-Skene-initialized Ising EM degrades. If both degrade together, the clique is an identifiability problem and not an optimization problem.
- Pruning test. Whether pruning by freezing temperature recovers full-panel accuracy with fewer judges than agreement-based pruning, especially when the clique judges are individually accurate.
- Calibration check. Whether the annealed fit's posterior has lower expected calibration error than the fixed-temperature fit, given the calibration trap documented for tabular ensembles.
References
- Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
- Deterministic Annealing Variant of the EM Algorithm
- Deterministic Annealing for Semi-Supervised Structured Output Learning
- ASAP: Architecture Search, Anneal and Prune
- Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
- Ensembling Tabular Foundation Models: A Diversity Ceiling and a Calibration Trap
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification
- Teaching Machines to Read and Comprehend