Abstract
Deep Neural Networks (DNNs) have achieved remarkable success across various domains, yet they frequently suffer from poor calibration, often exhibiting overconfidence in their predictions. In critical decision-making applications such as autonomous driving or medical diagnosis, reliable confidence scores are essential for system trust and safety. While Temperature Scaling (TS) is a popular, low-complexity calibration method, it falters when validation sets are small, labels are noisy, or models are highly accurate. This paper introduces Attended Temperature Scaling (ATS), a robust approach that improves calibration in these challenging scenarios while maintaining the efficiency of standard TS. We provide theoretical justifications and demonstrate the effectiveness of ATS across diverse models and datasets, including a practical application in skin lesion detection.
Introduction
In high-stakes environments, a model’s confidence must accurately reflect its probability of being correct. A well-calibrated model that claims 99% certainty should be correct 99% of the time. However, modern DNNs are notoriously overconfident.
The current state-of-the-art for post-hoc calibration is Temperature Scaling (TS), which rescales raw model outputs (logits) using a single learned parameter $T$. Despite its simplicity, TS has three primary failure modes:
- Data Scarcity: It struggles with small validation sets.
- Label Noise: Incorrect labels in the calibration set lead to skewed confidence levels.
- The Accuracy Paradox: Highly accurate models provide too few misclassified examples for TS to learn appropriate uncertainty.
Attended Temperature Scaling (ATS)
To address these limitations, we propose Attended Temperature Scaling (ATS). This method introduces two key innovations to the calibration process:
1. Smart Subset Construction ($M_k$)
Rather than treating the validation set as a monolithic block, ATS constructs class-specific subsets ($M_k$). These subsets include samples belonging to class $k$ and "attended" samples from other classes that the model nearly classified as $k$ (determined by a threshold $ heta$). This effectively increases the sample size for learning the optimal temperature, particularly for samples near the decision boundary where calibration is most critical.
2. A Specialized Loss Function ($L_{ATS}$)
Standard TS minimizes Negative Log Likelihood (NLL). ATS replaces this with a new loss function designed to treat calibration as a class-wise binary problem. This formulation is significantly more robust to label noise and helps the model better distinguish between correctly and incorrectly confident predictions.
Experimental Evaluation
We evaluated ATS against standard benchmarks and a critical real-world medical application:
- General Vision: CIFAR-10, CIFAR-100, SVHN, MNIST, and ImageNet2012.
- Fine-grained Classification: Caltech-UCSD Birds.
- Medical Diagnosis: The ISIC 2018 dataset for skin lesion analysis, featuring seven distinct lesion types.
Results and Impact
ATS consistently outperformed standard TS, Matrix Scaling, and Vector Scaling in terms of Expected Calibration Error (ECE) across various architectures (ResNet, DenseNet, VGG).
Robustness and Efficiency
- Noise Resilience: By focusing on conditional distributions rather than hard labels, ATS remains stable even when validation data contains errors.
- Data Efficiency: ATS maintains calibration performance with validation sets as small as 1,000 samples, where TS performance typically degrades.
- Preserving Performance: As a post-processing step, ATS modifies confidence scores without altering the model's predicted classes, ensuring no loss in accuracy.
Medical Referral Systems
In the context of skin lesion detection, we tested a referral system where low-confidence cases are sent to human specialists. ATS significantly widened the confidence gap between correct and incorrect predictions. This allowed the system to refer fewer correct cases while catching a higher percentage of potential misclassifications compared to uncalibrated models.
Conclusion
Attended Temperature Scaling (ATS) provides a robust and efficient solution for calibrating deep networks, particularly in data-constrained or noisy environments. By attending to samples near decision boundaries and employing a specialized loss function, ATS ensures that model confidence is a reliable metric for human-in-the-loop and autonomous decision-making systems.
Summary updated