Language models are now routinely used as judges that grade other models' outputs. A judge's overall agreement with humans can look fine while it fails badly on particular kinds of inputs. Diagnosing LLM Judge Reliability showed this on the SummEval summarization benchmark. It wrapped judge scores in conformal prediction, a distribution-free way to turn a point score into a set of plausible human scores with a guaranteed hit rate. The guarantee holds only on average over the whole dataset. The authors name coverage within localized subsets of documents as an unsolved limitation. A 90% average can hide regions where the judge's sets miss the human score far more often.
We propose to fix this with a self-organizing map, a classic neural method that lays a small 2D grid of prototype vectors over a high-dimensional data cloud so that neighboring grid cells hold similar data. We fit the map to cheap, black-box "behavior signatures" of the judge on each input. We then calibrate the judge's uncertainty separately in each map region, borrowing calibration examples from neighboring cells when a cell has too few human labels. The result is a per-region guarantee and a picture of where a given judge can and cannot be trusted. It needs no access to model weights.
Core Claim
Grid neighborhoods on a self-organizing map are a good way to share scarce human calibration labels across similar inputs. Calibrating conformal sets per map region, with lattice smoothing, should cut the worst-region coverage shortfall of marginal conformal prediction on SummEval. It should also keep sets narrow where the judge is reliable.
We make a second, independent claim about the input to the map. The map should organize instances by how the judge behaves on them, not only by what the text says. Behavior here means the spread of scores across repeated samples, sensitivity to swapping the order of two responses, and the embedding of the judge's written rationale. Two lines of work motivate this choice. Rating Roulette shows that judges disagree with themselves across repeated samples, and that setting temperature to zero hides this variation at a cost in human alignment. That makes the sampled spread an informative signal rather than noise. SOM Directions are Better than One shows that a self-organizing map over LLM activations captures structure that a single linear direction misses. There, refusal is spread across several prototype directions, which raised attack success on Llama-2-7B from 0.0% to 59.11%.
We borrow that paper's technique of using the map's prototypes as a multi-faceted summary of model behavior. We apply it to a different target, black-box judge outputs rather than internal activations, and to a different goal, calibration rather than ablation.
Research Program
For each document–summary pair and each quality criterion, we build a signature vector from four black-box parts:
- a histogram of ten scores sampled at temperature 1 on the 1–5 scale;
- the rate at which pairwise preferences against a few sibling summaries of the same document flip when the two responses swap positions;
- the mean embedding of the judge's sampled rationales;
- a sentence embedding of the source and summary.
The swap-flip measure uses the position bias that PandaLM, JudgeLM and FairJudge document as a per-instance feature.
We train a self-organizing map of roughly 8×8 to 12×12 cells on these signatures, using an unlabeled split kept disjoint from both calibration and test data. Each cell's prototype is its best-matching unit, the cell whose prototype is nearest to a given input.
The nonconformity score measures how far off the judge is on an instance: it is the gap between the human expert score and the judge's sampled score distribution. Each test point $x$ gets its best-matching cell $c(x)$. We compute a weighted quantile of calibration nonconformity scores $s_j$:
$$\hat q(x) = \text{Quantile}_{1-\alpha}\Big(\textstyle\sum_j \tilde h_{c(x),c(x_j)}\,\delta_{s_j} + \tilde h_{c(x),c(x)}\,\delta_{\infty}\Big).$$
In this expression, $\tilde h_{c,c'}$ is the Gaussian lattice kernel between cells $c$ and $c'$, $\exp(-\|r_c-r_{c'}\|^2/2\sigma^2)$, normalized over the points; $r_c$ is cell $c$'s grid coordinate. $\delta$ denotes a point mass, and $1-\alpha$ is the target coverage. With $\sigma \to 0$ this becomes hard per-cell calibration, which gives exact finite-sample coverage within each cell. That follows from the standard exchangeability argument, applied separately within each stratum. Larger $\sigma$ trades exactness for stability in sparse cells. We pick $\sigma$ per cell by requiring a minimum effective number of calibration points, and report the hard and smoothed versions separately.
The map also serves as an audit tool. We overlay three quantities on the grid: per-document 3-cycle transitivity violations (A preferred to B, B to C, yet C to A), empirical judge error, and rubric flip rates from Judging LLM-as-a-Judge. This tests whether these failures form contiguous regions. That paper visualized rubric geometry with a UMAP projection and BERTopic clusters, but it produced no reusable prototypes and did no calibration.
Baselines
The naive reference is the raw judge score reported as a point estimate. The first real baseline is marginal split conformal as in Diagnosing LLM Judge Reliability. On SummEval it reports about 90% marginal coverage, average set size ~3.0 on Relevance and ~4.9 on Fluency and Consistency, and a Spearman correlation of +0.576 between set width and human disagreement. Beating it means matching its marginal coverage with a smaller worst-group coverage gap, and without a larger average set size.
A stronger baseline stratifies conformal calibration by criterion and by the judge's modal score. That paper found difficulty depends heavily on criterion, so this cheap stratification may capture much of the heterogeneity on its own.
The strongest competitor is localized conformal prediction by k-nearest-neighbor weighting, run directly in the same signature space with no map. No reported result was found on Lacuna for kNN-localized conformal on LLM judges. We will run it with the same features, the same calibration labels, and the same budget for tuning $k$ or the bandwidth.
Two ablations isolate the proposal's ingredients. The first is k-means stratification with the same number of cells, which keeps the clustering and removes the topology. It follows the contrast drawn in The Comparison of SOM and K-means for Text Clustering. The second is the same map trained on text embeddings alone, which removes the judge-behavior signature.
Every method gets identical splits, identical judge calls, and a tuning grid of equal size, chosen on a validation split. The primary comparison is the regime with 200–400 human-labeled calibration instances, since that is where borrowing across neighbors should matter.
Why not just…?
Why not just use k-means clusters as calibration strata? Hard k-means strata give exact per-cluster coverage too. With few labels, however, many clusters are sparse, and k-means offers no principled neighbor to borrow from. That is exactly what the lattice kernel supplies. The k-means ablation, run with and without the lattice, measures how much of any gain comes from topology rather than from clustering alone.
Why not just do kNN-weighted localized conformal in the embedding space? This is a real threat, and it may match us on coverage. Our bet is that neighbors on the lattice approximate neighbors along the data manifold better than raw Euclidean kNN in a high-dimensional space. That advantage would appear mainly with small calibration sets. If kNN ties at every label budget, we will say so. The contribution then narrows to the behavior-signature features and the auditable map, which kNN does not produce: a fixed, inspectable set of prototypes that can be labeled "untrustworthy for judge J."
Why not just use the judge's own sample variance as the trust signal? Rating Roulette shows the variance exists, and we include it inside the signature. On its own, however, variance is a heuristic without a coverage guarantee. Rubric stickiness, where judges kept incorrect verdicts 62.3% of the time after the evidence was flipped, suggests judges can be confidently wrong. We report the variance-only ranking of instances as a baseline for predicting error, and add a conformal version stratified by variance quantiles.
Why not just fine-tune a distributional judge? Distributional LLM-as-a-Judge trains the judge to predict the full distribution of human labels. It reports an SNLI KL divergence of 0.23 against 0.60 for single-label training. That approach needs open weights and multi-annotator training data, and it gives no coverage guarantee. Our method targets closed API judges and could be layered on top of a distributional judge. We therefore treat that line as complementary rather than as a head-to-head baseline.
Proposed Experiments and Decisive Checks
The primary testbed is SummEval (Yale-LILY/SummEval), using its expert 1–5 annotations on all four criteria. We test the same judges as Gupta and Kumar: GPT-4o-mini, LLaMA-3.1-70B, Qwen-2.5-72B and Mistral-Small.
Conditional coverage is evaluated on groups defined independently of the map, to avoid circularity. The groups are source document, summarizer system, criterion and summary-length quartile. We use leave-documents-out splits so that a document's summaries never appear in both calibration and test.
The metrics are:
- marginal coverage;
- worst-group coverage;
- mean absolute coverage gap across groups;
- average set size;
- Spearman correlation of set width with error.
Each metric is swept over calibration budgets from 100 to the full set.
HealthBench rubric-level pass/fail judgments give a second, binary setting. There we also test whether map regions predict the rubric flip rates reported in Judging LLM-as-a-Judge.
Three results would settle the proposal:
- Coverage gap. Smoothed map calibration must cut the worst-group coverage gap of marginal conformal by a clear margin at a matched average set size, and must beat k-means strata at small budgets.
- Clustering of failures. Transitivity violations must be spatially clustered on the map, with lattice Moran's I, a spatial autocorrelation statistic, significant under permutation, and clustered more strongly on the behavior-signature map than on the text-only map. If violations are scattered uniformly, then the masked-heterogeneity failures are not organized by region, and the audit claim fails.
- kNN comparison. This decides whether topology earns its place, or whether the contribution reduces to the judge-behavior signature.
References
- Diagnosing LLM Judge Reliability: Conformal Prediction Sets and Transitivity Violations
- Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks
- SOM Directions Are Better than One: Multi-Directional Refusal Suppression in Language Models
- FairJudge : An Adaptive, Debiased, and Consistent LLM-as-a-Judge
- Judging LLM-as-a-Judge: Concerning Rubric Artifacts in LLM-based Automated Text Generation Evaluation
- The Comparison of SOM and K-means for Text Clustering
- Distributional LLM-as-a-Judge