Calibrating by Asking the Neighbors: Asymmetric Attention over Held-Out Errors

A classifier is calibrated when its stated confidence matches how often it is right. The standard fix, temperature scaling, divides every output score by one learned number. It fails in a specific way. Models are overconfident in some regions of input space and underconfident in others: overconfident in sparse regions, underconfident in dense ones, and wrong in a systematic direction wherever classes overlap. One number cannot correct errors that point in different directions at once. We propose a small drop-in module that sits on top of any frozen classifier and calibrates each prediction by consulting a labeled held-out set. The test example's internal representation acts as a query, a search signal. It attends over the stored representations of held-out examples, which act as keys. From the matched examples it reads back what the model got wrong on them: their label minus the model's predicted probabilities. The averaged local error becomes a correction for each class and each instance.

The design rests on two claims. The first is that held-out labels in a neighborhood reveal class overlap that training labels hide, because a network fit to its training set "picks a side" in an overlapping region on training data but cannot do so on unseen data. The second is that the matching should be asymmetric. The projection that turns a test example into a query should differ from the projection that turns a stored example into a key, because "which stored examples are evidence about this test point" is a directional question. Proximity is not reciprocal: an isolated point's nearest neighbors sit in a dense cluster, but that cluster's neighbors do not include the isolated point. Symmetric density estimates and nearest-neighbor rules ignore this direction.

Core Claim

A post-hoc, attention-based calibrator that aggregates signed held-out residuals through asymmetric query and key projections can repair three failures of global scaling. It can fix overconfidence in overlapping class regions, proximity bias, and class-wise under- and over-confidence, all without retraining the backbone. It should outperform both scalar instance-wise temperature methods and symmetric density-based recalibration. It should also close much of the gap to training-time fixes such as Mixup on controlled overlap problems. What is new is the combination of three ingredients: attention over stored, labeled error signals rather than over the input's own tokens; a correction for each class rather than a single temperature; and an explicit test of whether query–key asymmetry, rather than retrieval alone, carries the benefit.

Where This Sits

Three results motivate the proposal. On the Limitations of Temperature Scaling for Distributions with Overlaps proves that even an oracle temperature cannot fix overconfidence in overlapping regions without damaging calibration in separable ones. On a 300-dimensional overlapping Gaussian mixture it reports a negative log-likelihood (NLL) of 4.2977 for ERM plus temperature scaling against 0.7444 for 4-Mixup. Its remedy is a change at training time. Our claim is narrower: the impossibility applies to monotone global rescaling, not to corrections that differ by instance and draw on held-out labels. Proximity-Informed Calibration for Deep Neural Networks (PROCAL) shows that networks are most overconfident in low-density regions, and that global Expected Calibration Error (ECE) hides this because errors in dense and sparse regions cancel. Its fix uses a symmetric kernel density or bin-based estimate of proximity, a single scalar per input. Two Sides of Miscalibration shows that Vision Transformers and BERT are often underconfident, that a global temperature above one makes this worse for some classes, and it introduces the signed Miscalibration Score (confidence minus accuracy) and class-wise temperature scaling. Together these papers say the correction must be local, signed, and class-aware. No single existing method has all three properties.

The instance-wise temperature methods supply the post-hoc template we build on, and they also fix the capacity comparison our method must survive. Parameterized Temperature Scaling feeds sorted logits to a small network that predicts a temperature. Sample-Dependent Adaptive Temperature Scaling maps the typicality of a frozen feature, scored by a variational autoencoder, to a temperature. Calibration Attention (CalAttn) attaches a two-layer network to a Vision Transformer's summary token and trains it jointly with the backbone. All three output one scalar per instance, so the predicted class ranking is preserved but probability mass across the remaining classes is not redistributed. None of them looks at held-out labels at test time. Feature Clipping shows that badly calibrated samples carry unusually large feature magnitudes, which suggests feature geometry carries the signal. It applies one global clipping threshold.

The asymmetric-attention literature supplies the matching mechanism and a key constraint. Primal-Attention recasts self-attention as an asymmetric kernel singular value decomposition with separate query-side and key-side factors. KEP-SVGP and Revisiting Kernel Attention with Correlated Gaussian Process Representation (CGPT) build uncertainty-aware attention while keeping query and key projections distinct. CGPT specifically shows that tying them, as earlier Gaussian-process attention did, costs both capacity and calibration on CIFAR-10-C. Both, however, rebuild the attention layers of the backbone and train end to end. They place asymmetry inside the model's self-attention, not in a calibrator that queries labeled evidence. We take their lesson, that tying query and key hurts, and turn it into a falsifiable ablation in a new location.

The modular-integration work supplies the deployment constraint. Pose-aware attention blocks and LOTFormer show that attention modules which keep the input and output tensor shapes of the layer they wrap can be dropped into pretrained backbones with little disruption. Our module keeps the classifier's contract: logits in, logits of the same shape out. It therefore applies to convolutional networks, Vision Transformers, and BERT alike. One tension deserves mention. LOTFormer argues for balancing attention so that rows and columns both sum to one. Our setting argues the opposite: some stored examples, the ones in overlap regions, should be consulted by many queries, and isolated queries should concentrate on a few keys. We therefore keep ordinary row-wise softmax and treat the balanced variant as a secondary comparison.

Research Program

Let $h(x)$ be the frozen penultimate feature of input $x$, $z(x)$ its logits over $K$ classes, and $p(x)$ its softmax probabilities. A held-out labeled set is split into a bank $\{(h_j, y_j, p_j)\}$ and a disjoint fitting set. Each bank item stores a residual $e_j = \mathrm{onehot}(y_j) - p_j$, the signed error the model made on it. The module computes

$$r(x) = \sum_{j \in \mathcal{N}(x)} a_j(x)\, \big[e_j \,\Vert\, W_V h_j\big], \qquad a_j(x) = \mathrm{softmax}_j\!\left(\frac{(W_Q h(x))^\top (W_K h_j)}{\sqrt{d}} + \beta\, \phi(z(x), z_j)\right),$$

where $\mathcal{N}(x)$ is a top-$k$ candidate set retrieved for speed, $W_Q \neq W_K$ are the query and key projections into dimension $d$, $W_V$ is a value projection, $\Vert$ denotes concatenation, and $\phi$ is an optional logit-agreement term with learned weight $\beta$. A two-layer head maps $r(x)$, together with the sorted logits, to an output. The default output is order-preserving: an instance-specific temperature plus monotone per-rank adjustments of the sorted logits, so accuracy is unchanged. A second variant applies unconstrained per-class logit shifts and reports any accuracy change. The module is fit by minimizing NLL plus a Brier term on the fitting set. Roles are cross-fit, with bank and fitting halves swapped and averaged, so that no item attends to itself.

The intended behavior is concrete. In an overlap region, the bank residuals around $x$ average to "the model said 0.95 for class A, but held-out labels were split," and the module softens the prediction. In a dense, easy region the residuals are slightly positive for the predicted class, indicating underconfidence, and the module sharpens. For an isolated query, the attention weights and the softmax temperature it learns should reflect that its evidence is distant. We will verify that by reading the attention entropy against PROCAL's proximity score.

The analysis has three parts. First, we measure whether the learned $W_Q$ and $W_K$ actually diverge, using the principal angles between their row spaces. We also check whether the divergence correlates with the density gap between query and key, which is the mechanism we claim. Second, we test whether multi-depth queries help. Here the query is built from features of the last few blocks, in the spirit of BA-Net's pooling across layers, while the keys stay penultimate. This part is secondary and is dropped if it does not help. Third, we study how bank size affects performance, because data scarcity is the known weak point of higher-capacity calibrators.

Baselines

The naive reference points are the uncalibrated model and global temperature scaling fit on the same held-out set. Matching the held-out budget is the whole fairness question here. Every post-hoc method gets exactly the same held-out examples: our method splits them into bank and fitting halves, while competitors use all of them for fitting. A useful anchor is Feature Clipping, which reports ECE on CIFAR-100 with ResNet-50 dropping from 17.52% uncalibrated to 5.72% with temperature scaling and 3.98% with clipping. Beating temperature scaling is required but not interesting.

The strongest scalar instance-wise competitors are Parameterized Temperature Scaling and AdaTS, run on the same frozen features with their published architectures. They also run at a capacity-matched size, scaled up to our module's parameter count, so that any gain cannot be attributed to parameter count alone. AdaTS reports CIFAR-100 WideResNet ECE falling from 3.76% with temperature scaling to 2.95% Sample-Dependent Adaptive Temperature Scaling. We count a win only if we beat the capacity-matched version, not just the published one.

The strongest density-aware competitor, and the most direct threat, is PROCAL, run with its kernel-density and bin-mean-shift variants on the same features and held-out data. It reports iNaturalist 2021 ResNet-50 ECE falling from 4.85 to 0.78 Proximity-Informed Calibration. Beating it requires lower proximity-informed ECE (PIECE) and NLL at matched ECE, since its ECE number is already low. The class-wise competitor is class-wise temperature scaling from Two Sides of Miscalibration, which reports ImageNet ConvNext ECE falling from 3.60% to 1.30%. A win there means lower worst-class signed miscalibration, not just lower ECE.

Two reference points use more compute or retraining. CalAttn reports ImageNet-1K Swin-S ECE falling from 9.95% to 1.25% Calibration Attention, but it retrains the backbone. We report it as a ceiling and do not need to beat it to succeed post hoc. We would claim a strong result if we match it without touching backbone weights. Mixup retraining serves the same ceiling role on the overlap experiments.

Three ablations isolate the key ingredients. Tying $W_Q = W_K$ isolates asymmetry. Replacing the residual values $e_j$ with feature values only isolates the use of held-out errors, and in this form the module reduces to a retrieval-augmented temperature predictor. A plain $k$-nearest-neighbor average of residuals in raw feature space, with no learned projections, isolates learned attention altogether.

Why not just…?

Why not just use PROCAL or a nearest-neighbor residual average? This is the real threat. If a fixed-metric neighbor average of held-out residuals performs as well as our learned asymmetric version, then the attention adds nothing and the honest contribution is the residual-averaging idea alone. PROCAL's published form outputs a density-based adjustment that is symmetric and scalar, so it cannot express class-specific direction. The plain-neighbor ablation is built to settle this. We commit to reporting it as the headline comparison. If it matches, we will reframe the paper around signed residual retrieval and drop the asymmetry claim.

Why not just give PTS or AdaTS a bigger network? Their bottleneck, per Parameterized Temperature Scaling, is expressive power. But a scalar temperature computed from the instance's own features cannot know whether held-out points near it were mislabeled by the model. That information lives in labels the model never saw for that region. The capacity-matched runs test this directly. The synthetic overlap task is where the gap should be largest, because there the feature-only signal is ambiguous by construction.

Why not just fix it at training time with Mixup or CalAttn? These work. Mixup's NLL of 1.77 against 5.95 for scaled ERM on CIFAR-10 with 50% label noise On the Limitations of Temperature Scaling is the clearest evidence. But they require retraining the backbone, which is often unavailable for large pretrained models. They also do not adapt when a new labeled held-out set arrives. Our claim is that the post-hoc module closes a large fraction of this gap, and we report that fraction explicitly rather than claim parity.

Why not just use class-wise temperatures? Class-wise scaling Two Sides of Miscalibration fixes bias per class but is still global within a class. It cannot separate a confusable "cat" from an easy "cat." Measuring proximity-stratified signed miscalibration within each class will show whether this residual structure exists and whether our module removes it.

Proposed Experiments

The decisive first experiment is the 300-dimensional overlapping Gaussian mixture from On the Limitations of Temperature Scaling. It is followed by CIFAR-10 with 50% label noise. Here ERM plus temperature scaling (NLL 4.2977 on the mixture at severe overlap) and Mixup (0.7444) bracket the possible outcomes. If the post-hoc module on a frozen ERM model does not land well below 4.2977, the central hypothesis fails and we stop.

Real-data evaluation uses CIFAR-100 with ResNet-50 and WideResNet, ImageNet-1K with ConvNext and Swin-S, and iNaturalist 2021 with ResNet-50 for long-tailed sparsity. Shift robustness uses CIFAR-10-C and ImageNet-C, with the bank always drawn from clean in-distribution data so that the shift is honest. A language check uses SST-2 and TweetEval with BERT, where underconfidence was documented. The metrics are ECE, PIECE, NLL, Brier score, and the worst-class and worst-proximity-bin signed Miscalibration Score. Accuracy is reported for the unconstrained variant only.

Two further checks determine whether the paper's specific claims survive. The tied-projection ablation must lose by a consistent margin across backbones, and the divergence between $W_Q$ and $W_K$ must track the density gap between query and key; otherwise asymmetry is not the mechanism. The residual-free ablation must lose on overlap-heavy data; otherwise held-out errors are not the mechanism. Finally, sweeping bank size from a few hundred to the full held-out split tells practitioners when the module is worth using over temperature scaling. We expect the advantage to vanish at small banks, and we will report where that crossover falls.