Image classifiers trained on long-tailed data, where a few common classes supply most of the examples and most classes are rare, are badly miscalibrated. Their confidence scores do not match how often they are right. The standard fixes push one dataset-wide number, expected calibration error, close to zero. That number averages over everything. A rare-species classifier can therefore look calibrated while it says "80% sure" about two very different kinds of image. In one kind it is looking at the animal and is usually right. In the other it is looking at the water or grass that happened to surround every training photo of that species, and it is often wrong. The calibration literature calls this hidden spread grouping loss: inputs that receive the same confidence score but have genuinely different chances of being correct. Global recalibration provably cannot remove it.
This note proposes using the model's own explanation of each prediction as the signal that separates these groups. The explanation here is a gradient-based saliency map, a heatmap obtained by backpropagating the class score to the last convolutional layer to show which image regions drove the decision. The claim has two parts. First, a few summary statistics of the saliency map recover a large share of tail-class grouping loss that confidence scores and embedding-space neighborhoods miss: how concentrated the map is, how spatially coherent it is, and how much of its energy falls off the object. Second, these statistics do not depend on the class. A small recalibration model trained on them can therefore learn on data-rich head classes and transfer to tail classes, where neighborhood-based local calibration runs out of samples.
Core Claim and What Is New
Under a proper scoring rule such as the Brier score, the expected loss splits into three parts. Calibration loss is the gap between confidence and accuracy at each confidence level. Grouping loss is the spread of true correctness probability among inputs that share a confidence score. Irreducible loss is the noise inherent in the task. Writing $S$ for the model's top-class confidence, $Y$ for the event that the prediction is correct, and $Q = P(Y=1 \mid X)$ for the true per-input correctness probability, grouping loss is
$$ \mathrm{GL} = \mathbb{E}\big[(Q - \mathbb{E}[Q \mid S])^2\big], $$
which is exactly the error that no function of $S$ alone can remove. Any reduction must condition on something beyond $S$. We propose a recalibrator that conditions on saliency statistics:
$$ \hat p(x) = g\big(S(x), a(x)\big), $$
where $a(x)$ is a short vector of summary statistics of the predicted class's Grad-CAM map for input $x$. The function $g$ is a small monotone-in-$S$ model, for example a gradient-boosted tree or a two-layer network, shared across all classes and fit on a held-out split.
The prior work falls into three groups, and our proposal differs from each. The grouping-loss work motivates the proposal and supplies the estimator. Beyond calibration: estimating the grouping loss of modern neural networks builds a lower-bound estimator by fitting decision trees on penultimate-layer features within confidence bins. It shows that Platt scaling and isotonic regression leave grouping loss untouched, but it studies only balanced data and generic shift. Reconfidencing LLMs from the Grouping Loss Perspective applies the same tree partitioning to language-model hidden states and fits per-subgroup isotonic regression. It also shows that global calibration can raise grouping loss, from 0.95 to 1.14 on one relation for Mistral-7B. Both define subgroups from embedding geometry. We define them from attribution, and we test directly whether attribution carries information that embeddings do not.
The local-calibration methods constrain the design. Local Calibration: Metrics and Recalibration (LoRe) and Proximity-Informed Calibration for Deep Neural Networks recalibrate using feature-space neighbors or local density. The research file notes that such kernel estimates become very noisy exactly where tail classes live. This is why our recalibrator uses class-agnostic features rather than class-specific neighborhoods.
The long-tailed attribution work establishes the mechanism we exploit but leaves out confidence entirely. Shortcut-Resistant CAM Distillation for Long-Tailed Recognition (SRCD) shows that tail classes learn background shortcuts. It distills head-class activation maps into tail classes, scoring maps with a spatial-coherence term and an evidence-concentration term, and reports only accuracy and foreground attention. SMCL: Saliency Masked Contrastive Learning for Long-tailed Recognition uses an external saliency detector to reassign backgrounds to tail classes. Neither measures calibration.
The long-tailed calibration methods, Improving Calibration for Long-Tailed Recognition (MiSLAS), Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective (UniMix with Bayes-bias compensation), and Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution, all validate with global calibration error alone. Our work sits on top of these models rather than replacing them.
Research Program
The first stage is a measurement study. We take standard long-tailed models: cross-entropy training, the margin-based LDAM-DRW recipe from Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, MiSLAS, and an SRCD-trained model. Each is first globally recalibrated, and we then estimate grouping loss separately for head, medium and few-shot classes.
The estimator is fit three ways: with partitions on penultimate embeddings only, on saliency statistics only, and on both. If saliency-based partitions reveal grouping loss that embedding partitions miss, concentrated in few-shot classes, the first half of the claim holds.
There is a methodological trap here. An estimator that partitions on embeddings is biased toward finding embedding-defined structure. We therefore always report grouping loss estimated on the union of features, with partitions fit on one split and evaluated on another. We also report calibration within ground-truth shortcut groups wherever we can construct them (described below), since that needs no estimator at all.
The saliency vector $a(x)$ has two variants. The mask-free variant contains Grad-CAM entropy, the SRCD-style coherence and concentration scores, the fraction of map energy in the image border, and the map's agreement with maps for the second-ranked class. It is usable at deployment because it needs no segmentation. The mask-based variant adds the Grad-CAMO overlap score from Grad-CAMO: Learning Interpretable Single-Cell Morphological Profiles from 3D Cell Painting Images, the fraction of attribution energy inside the object mask. It is used only to validate that the mask-free statistics track real foreground reliance.
Gradients saturate when the model is extremely confident. We therefore compute Grad-CAM on pre-softmax scores. As a robustness check, we replace Grad-CAM with Integrated Gradients, which integrates gradients along a path from a neutral baseline image to the input and so avoids that saturation.
The second stage builds and evaluates the recalibrator $g$. It changes only the top-class confidence, so accuracy is untouched and any gain is purely in reliability. The ingredient that matters is sharing across classes. We fit $g$ on held-out data pooled over all classes and compare it against per-class and per-frequency-group versions. If pooling helps tail classes most, the head-to-tail transfer argument holds.
Baselines
The naive reference is the plain cross-entropy model with global temperature scaling: one scalar dividing the logits, fit on validation data. This model already scores well on global calibration error, and grouping-loss theory says it should leave grouping loss unchanged. Every method is reported against it.
The strongest training-time competitor is MiSLAS, which applies stronger label smoothing to head classes than to tail classes and also adjusts batch-normalization statistics. On CIFAR-100-LT with imbalance factor 100, it reports reducing global calibration error from 27.5% to 4.83% at 47.0% accuracy (Improving Calibration for Long-Tailed Recognition). The strongest post-hoc competitor is head-to-tail Gaussian transfer with importance-weighted temperature scaling. On ImageNet-LT it reports a global calibration error of 3.45%, against 6.72% for standard temperature scaling (Transfer Knowledge from Head to Tail). We found no reported grouping-loss or local calibration error results on Lacuna for any long-tailed benchmark. We will therefore re-run both methods under our grouping-loss evaluation, using their published recipes and the same backbones, data splits and calibration split as our method.
Three embedding-based local recalibrators are the closest methodological competitors. The first is tree-based latent reconfidencing: subgroups from trees on penultimate features, then isotonic regression within each subgroup, following Reconfidencing LLMs from the Grouping Loss Perspective. The second is LoRe and the third is the density-aware method PROCAL. All three receive the same held-out data and the same hyperparameter search budget as $g$.
We will also run class-wise temperature scaling and temperature scaling fit separately per frequency group (head, medium, few-shot), because these are the simplest form of conditioning beyond $S$.
The decisive ablation keeps the architecture and capacity of $g$ fixed and swaps its inputs. We compare confidence only, confidence plus a dimension-matched projection of the embedding, confidence plus saliency statistics, and confidence plus saliency statistics shuffled across images. Beating the baselines means lowering few-shot grouping loss and few-shot Brier score while matching their global calibration error, with the improvement holding across three seeds and disappearing under shuffling.
Why not just…?
Why not just fit temperature scaling per class or per frequency group? This removes calibration differences between classes. However, a single per-class temperature is still a monotone function of $S$ within each class, so it cannot separate a confident foreground prediction from an equally confident background prediction of the same class. The grouping-loss analysis in Beyond calibration explains why. We include both variants as baselines. Our injected-shortcut test (below) shows directly whether they leave the within-class gap between shortcut and non-shortcut images.
Why not just partition penultimate embeddings, as latent reconfidencing does? This is the real threat. If shortcut reliance is linearly visible in the embedding, trees on embeddings will find it and saliency adds nothing. The counter-argument has two parts. Saliency statistics are a few class-agnostic numbers, whereas embedding subgroups are high-dimensional and tied to particular classes, which matters when a tail class has a handful of calibration examples. And SRCD's finding that shortcuts persist under strong contrastive training suggests that embeddings mix object and context. Neither part is guaranteed, so the matched-capacity ablation settles the question. If confidence-plus-embedding equals confidence-plus-saliency on few-shot grouping loss, we will report the claim as false.
Why not just remove the shortcut during training, with SRCD or SMCL, and then calibrate globally? If training eliminated background reliance, the saliency-defined groups would collapse and global calibration would suffice. But SRCD reports only that the foreground-to-background attention ratio roughly doubles on Pascal VOC 2012, not that background reliance disappears. Its forced localization might also create new overconfident errors. We apply our measurement and our recalibrator to an SRCD-trained model. Remaining grouping loss there that saliency can explain would show the two approaches are complementary. No remaining grouping loss would mean shortcut-resistant training alone is the better answer.
Proposed Experiments
The cleanest test uses injected shortcuts. On CIFAR-100-LT at imbalance factors 10, 50 and 100, we add a class-correlated background texture to most training images of each tail class. At test time, half the images carry the texture and half do not. This creates ground-truth groups whose calibration gap we can compute without any estimator. Success means three things: the saliency statistics predict group membership, the per-group calibration gap shrinks under $g$ but not under global or class-wise temperature scaling, and the gap stays under embedding-tree reconfidencing.
For mask-based validation, we build a long-tailed classification split of Pascal VOC 2012 from its object masks. This lets us check that the mask-free statistics correlate with Grad-CAMO overlap and that the mask-based and mask-free versions of $g$ perform similarly.
The natural-shortcut evaluation runs on ImageNet-LT, Places-LT and iNaturalist 2018. We use ResNet backbones matching the cited recipes and report the following measures broken out by head, medium and few-shot classes: global calibration error, local calibration error, the grouping-loss lower bound, Brier score and negative log-likelihood. We also report the area under the risk-coverage curve on few-shot classes, which measures how well confidence ranks predictions for selective abstention.
As an out-of-domain check, we apply the same audit to the 3D Cell Painting data, where Grad-CAMO found that only 30% of profiles overlapped cell boundaries despite 85% accuracy. We test whether that overlap explains grouping loss in a dose classifier.
The proposal fails if, on the natural benchmarks, saliency-conditioned recalibration does not beat matched-capacity embedding conditioning on few-shot grouping loss. In that case the honest conclusion is that attribution is a useful audit tool but not a calibration signal.
References
- Beyond calibration: estimating the grouping loss of modern neural networks
- Reconfidencing LLMs from the Grouping Loss Perspective
- Local Calibration: Metrics and Recalibration
- Proximity-Informed Calibration for Deep Neural Networks
- Shortcut-Resistant CAM Distillation for Long-Tailed Recognition
- SMCL: Saliency Masked Contrastive Learning for Long-tailed Recognition
- Improving Calibration for Long-Tailed Recognition
- Towards Calibrated Model for Long-Tailed Visual Recognition from Prior Perspective
- Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution
- Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss
- Grad-CAMO: Learning Interpretable Single-Cell Morphological Profiles from 3D Cell Painting Images