A New Loss Function for Temperature Scaling to have Better Calibrated Deep Networks

Azadeh Sadat Mozafari, Hugo Siqueira Gomes, Steeven Janny, Christian Gagné

Abstract

Deep Neural Networks (DNNs) have achieved remarkable success across various domains, yet they frequently suffer from poor calibration, often exhibiting overconfidence in their predictions. In critical decision-making applications such as autonomous driving or medical diagnosis, reliable confidence scores are essential for system trust and safety. While Temperature Scaling (TS) is a popular, low-complexity calibration method, it falters when validation sets are small, labels are noisy, or models are highly accurate. This paper introduces Attended Temperature Scaling (ATS), a robust approach that improves calibration in these challenging scenarios while maintaining the efficiency of standard TS. We provide theoretical justifications and demonstrate the effectiveness of ATS across diverse models and datasets, including a practical application in skin lesion detection.

Introduction

In high-stakes environments, a model’s confidence must accurately reflect its probability of being correct. A well-calibrated model that claims 99% certainty should be correct 99% of the time. However, modern DNNs are notoriously overconfident.

The current state-of-the-art for post-hoc calibration is Temperature Scaling (TS), which rescales raw model outputs (logits) using a single learned parameter $T$. Despite its simplicity, TS has three primary failure modes:

  1. Data Scarcity: It struggles with small validation sets.
  2. Label Noise: Incorrect labels in the calibration set lead to skewed confidence levels.
  3. The Accuracy Paradox: Highly accurate models provide too few misclassified examples for TS to learn appropriate uncertainty.

Attended Temperature Scaling (ATS)

To address these limitations, we propose Attended Temperature Scaling (ATS). This method introduces two key innovations to the calibration process:

1. Smart Subset Construction ($M_k$)

Rather than treating the validation set as a monolithic block, ATS constructs class-specific subsets ($M_k$). These subsets include samples belonging to class $k$ and "attended" samples from other classes that the model nearly classified as $k$ (determined by a threshold $ heta$). This effectively increases the sample size for learning the optimal temperature, particularly for samples near the decision boundary where calibration is most critical.

2. A Specialized Loss Function ($L_{ATS}$)

Standard TS minimizes Negative Log Likelihood (NLL). ATS replaces this with a new loss function designed to treat calibration as a class-wise binary problem. This formulation is significantly more robust to label noise and helps the model better distinguish between correctly and incorrectly confident predictions.

Experimental Evaluation

We evaluated ATS against standard benchmarks and a critical real-world medical application:

  • General Vision: CIFAR-10, CIFAR-100, SVHN, MNIST, and ImageNet2012.
  • Fine-grained Classification: Caltech-UCSD Birds.
  • Medical Diagnosis: The ISIC 2018 dataset for skin lesion analysis, featuring seven distinct lesion types.
Example of a skin lesion from the ISIC 2018 dataset used for calibration testing.

Results and Impact

ATS consistently outperformed standard TS, Matrix Scaling, and Vector Scaling in terms of Expected Calibration Error (ECE) across various architectures (ResNet, DenseNet, VGG).

Robustness and Efficiency

  • Noise Resilience: By focusing on conditional distributions rather than hard labels, ATS remains stable even when validation data contains errors.
  • Data Efficiency: ATS maintains calibration performance with validation sets as small as 1,000 samples, where TS performance typically degrades.
  • Preserving Performance: As a post-processing step, ATS modifies confidence scores without altering the model's predicted classes, ensuring no loss in accuracy.

Medical Referral Systems

In the context of skin lesion detection, we tested a referral system where low-confidence cases are sent to human specialists. ATS significantly widened the confidence gap between correct and incorrect predictions. This allowed the system to refer fewer correct cases while catching a higher percentage of potential misclassifications compared to uncalibrated models.

Dermoscopic image of a skin lesion, illustrating the complexity of the medical diagnostic task.

Conclusion

Attended Temperature Scaling (ATS) provides a robust and efficient solution for calibrating deep networks, particularly in data-constrained or noisy environments. By attending to samples near decision boundaries and employing a specialized loss function, ATS ensures that model confidence is a reliable metric for human-in-the-loop and autonomous decision-making systems.

Summary updated

Abstract

Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliability to the system. To calibrate the deep networks' confidence, many probabilistic and measure-based approaches are proposed. Temperature Scaling (TS) is a state-of-the-art among measure-based calibration methods which has low time and memory complexity as well as effectiveness. In this paper, we study TS and show it does not work properly when the validation set that TS uses for calibration has small size or contains noisy-labeled samples. TS also cannot calibrate highly accurate networks as well as non-highly accurate ones. Accordingly, we propose Attended Temperature Scaling (ATS) which preserves the advantages of TS while improves calibration in aforementioned challenging situations. We provide theoretical justifications for ATS and assess its effectiveness on wide range of deep models and datasets. We also compare the calibration results of TS and ATS on skin lesion detection application as a practical problem where well-calibrated system can play important role in making a decision.
BibTeX
@misc{mozafari2018new,
  author = {Azadeh Sadat Mozafari and Hugo Siqueira Gomes and Steeven Janny and Christian Gagné},
  title = {{A New Loss Function for Temperature Scaling to have Better Calibrated Deep Networks}},
  year = {2018},
  eprint = {1810.11586},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1810.11586},
}
Figure