Multiclass Local Calibration With the Jensen-Shannon Distance

Cesare Barbera, Lorenzo Perini, Giovanni De Toni, Andrea Passerini, Andrea Pugnana

Abstract

Developing trustworthy Machine Learning (ML) models requires their predicted probabilities to be well-calibrated, meaning they should reflect true-class frequencies. Among calibration notions in multiclass classification, strong calibration is the most stringent, as it requires all predicted probabilities to be simultaneously calibrated across all classes. However, existing approaches to multiclass calibration lack a notion of distance among inputs, which makes them vulnerable to proximity bias: predictions in sparse regions of the feature space are systematically miscalibrated. This is especially relevant in high-stakes settings, such as healthcare, where the sparse instances are exactly those most at risk of biased treatment. In this work, we address this main shortcoming by introducing a local perspective on multiclass calibration. First, we formally define multiclass local calibration and establish its relationship with strong calibration. Second, we theoretically analyze the pitfalls of existing evaluation metrics when applied to multiclass local calibration. Third, we propose a practical method for enhancing local calibration in Neural Networks, which enforces alignment between predicted probabilities and local estimates of class frequencies using the Jensen-Shannon distance. Finally, we empirically validate our approach against existing multiclass calibration techniques.

Introduction

In high-stakes fields like medicine, it isn't enough for a model to be accurate; it must be calibrated. If a model says there is a 70% chance a cell is cancerous, that should mean that out of 100 similar cases, 70 are actually cancerous. Most existing calibration techniques are "global"—they look at the model's performance across the entire dataset. However, this can hide proximity bias. A model might be perfectly calibrated on average but wildly overconfident in "sparse" regions of the feature space where it hasn't seen many examples. This is dangerous in healthcare, where rare or "edge-case" patients are often the ones most at risk of biased treatment. This paper introduces Multi-Class Local Calibration, a new framework that ensures machine learning models are reliable not just on average, but across different regions of the feature space.

Methodology: LoCal Nets (LCNs)

The authors propose a two-pronged approach: a new theoretical framework and a practical implementation called LoCal Nets (LCNs).

Theoretical Framework

They formally define multiclass local calibration, requiring a model’s predicted probability vector to align with the local empirical class frequencies (estimated via kernels). They prove that achieving local calibration is a sufficient condition for "strong calibration" (alignment across all classes simultaneously).

Implementation

Unlike standard post-hoc calibrators (like Temperature Scaling) that simply rescale the final outputs (logits), LCNs reshape the feature representation itself.

LCN flow compared to existing post-hoc calibration methods

As shown in the architecture comparison, LCNs learn a new, reduced-dimensionality feature space $\phi'(x)$. They use the Jensen-Shannon Distance (JSD) to minimize the difference between the model's predicted probabilities and the local class frequencies of neighboring points. Additionally, a similarity loss (Categorical Cross Entropy) ensures that points with the same labels are pulled closer together in this new feature space.

Key Insights and Results

The authors evaluated their approach on three diverse multiclass datasets: CIFAR-10, CIFAR-100, and TissueMNIST (a medical imaging dataset with significant class imbalance).

Reshaping vs. Rescaling

LCNs demonstrate that by actually changing how the model "sees" the geometry of the data (the feature space), you can achieve much tighter clusters and more reliable local predictions. Standard metrics like Expected Calibration Error (ECE) can suffer from "cancellation effects," where overconfidence in one region is masked by underconfidence in another. Local metrics (LCE) are necessary to see the true picture.

Visualizing the effect of LCN on CIFAR-10 feature representations and local calibration metrics

Performance Gains

  • Massive Reductions in Local Error: On CIFAR-10, LoCal Nets achieved a ~64% reduction in Maximum Local Calibration Error (MLCE), dropping from approximately .999 to .640, while Local Calibration Error (LCE) fell from .025 to .009.
  • Outperforming State-of-the-Art: LCNs consistently outperformed strong baselines like Dirichlet Calibration (DC) and Temperature Scaling (TS) on local metrics across all datasets.
  • Predictive Gains: Interestingly, LCNs actually improved model accuracy (e.g., +2.7% on TissueMNIST). This suggests that learning a better-calibrated feature space also helps the model classify better.

Limitations and Future Work

While promising, the approach has some limitations. Theoretical consistency proofs hold only as the sample size approaches infinity; in finite-sample regimes, kernel estimates can still suffer from bias. Additionally, the kernel bandwidth ($\gamma$) requires careful tuning to balance noise and locality. Future work may explore adaptive kernels that change their size based on data density, which could further mitigate proximity bias in very sparse regions.

Summary updated

Abstract

Developing trustworthy Machine Learning (ML) models requires their predicted probabilities to be well-calibrated, meaning they should reflect true-class frequencies. Among calibration notions in multiclass classification, strong calibration is the most stringent, as it requires all predicted probabilities to be simultaneously calibrated across all classes. However, existing approaches to multiclass calibration lack a notion of distance among inputs, which makes them vulnerable to proximity bias: predictions in sparse regions of the feature space are systematically miscalibrated. This is especially relevant in high-stakes settings, such as healthcare, where the sparse instances are exactly those most at risk of biased treatment. In this work, we address this main shortcoming by introducing a local perspective on multiclass calibration. First, we formally define multiclass local calibration and establish its relationship with strong calibration. Second, we theoretically analyze the pitfalls of existing evaluation metrics when applied to multiclass local calibration. Third, we propose a practical method for enhancing local calibration in Neural Networks, which enforces alignment between predicted probabilities and local estimates of class frequencies using the Jensen-Shannon distance. Finally, we empirically validate our approach against existing multiclass calibration techniques.
BibTeX
@inproceedings{barbera2026multiclass,
  author = {Cesare Barbera and Lorenzo Perini and Giovanni De Toni and Andrea Passerini and Andrea Pugnana},
  title = {{Multiclass Local Calibration With the Jensen-Shannon Distance}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2026},
  url = {https://openreview.net/forum?id=atUXOpyCox},
}
Figure