Abstract
Modern neural networks often suffer from miscalibration, exhibiting overconfidence in their predictions. While training with Focal Loss—controlled by a hyperparameter $\gamma$—has been shown to improve calibration compared to standard cross-entropy, existing approaches typically use a fixed or heuristic-based $\gamma$. We propose AdaFocal, a calibration-aware adaptive focal loss that dynamically modifies $\gamma$ for different groups of samples based on the model's performance on a validation set. By utilizing both focal and inverse-focal loss, AdaFocal addresses both under- and over-confidence. Our evaluations across various image recognition and NLP tasks demonstrate that AdaFocal significantly improves calibration and out-of-distribution detection while maintaining competitive accuracy.
1. Introduction
In many real-world applications, from autonomous driving to medical diagnosis, it is crucial that a neural network's confidence scores reflect the actual probability of being correct. This property is known as calibration. Unfortunately, modern deep neural networks are notoriously poorly calibrated, often producing highly overconfident predictions even when they are wrong.
Recent research highlighted that the Focal Loss, originally designed to handle class imbalance in object detection, acts as a regularizer that improves calibration by penalizing overconfident predictions. This effect is governed by the parameter $\gamma$. However, a fixed $\gamma$ is often insufficient, as different samples may require different levels of regularization. While sample-dependent heuristics like FLSD-53 have been proposed, they often lack generalization. AdaFocal addresses this by introducing a principled, adaptive approach to parameter selection.
2. The AdaFocal Approach
AdaFocal (Calibration-aware Adaptive Focal Loss) treats $\gamma$ as a dynamic parameter that evolves during training based on real-time feedback from the model's performance.
2.1 Validation-Driven Feedback and Binning
To determine how to adjust $\gamma$, AdaFocal monitors the model's calibration on a validation set. The probability space is divided into several "confidence bins" (e.g., 15 bins). For each bin, the algorithm calculates whether the model is currently overconfident (accuracy < confidence) or under-confident (accuracy > confidence).
2.2 Adaptive Update Rule
Instead of a global setting, AdaFocal maintains specific $\gamma$ values for different groups of samples. The update rule for $\gamma_t$ at step $t$ depends on the value from the previous step $\gamma_{t-1}$ and the observed calibration error in the corresponding bin. If a bin shows overconfidence, $\gamma$ is increased to increase regularization; if it shows under-confidence, $\gamma$ is decreased.
2.3 Inverse-Focal Loss
A key innovation in AdaFocal is the use of Inverse-Focal Loss. While standard Focal Loss (positive $\gamma$) reduces the weight of easy, high-confidence samples to curb overconfidence, AdaFocal allows $\gamma$ to become negative. This "inverse" application pushes the scores of under-confident samples higher, ensuring the model is not excessively "shy" about its correct predictions.
3. Experimental Evaluation
The researchers evaluated AdaFocal across diverse benchmarks to ensure its robustness:
- Computer Vision: Experiments were conducted on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet using architectures like ResNet, WideResNet, and DenseNet.
- Natural Language Processing: The method was tested on the 20 Newsgroups dataset using both CNNs and fine-tuned BERT models.
- Out-of-Distribution (OOD) Detection: The models were tested on SVHN and corrupted datasets (CIFAR-10-C) to evaluate their ability to identify unfamiliar data.
4. Key Results
4.1 Superior Calibration
AdaFocal consistently achieved the lowest Expected Calibration Error (ECE) across nearly all tested configurations. For instance, on CIFAR-100 with a ResNet-50 architecture, AdaFocal reduced the ECE from 17.17% (cross-entropy) to a mere 1.36%.
4.2 Accuracy and OOD Performance
Crucially, AdaFocal improves calibration without sacrificing predictive performance. In many cases, top-1 accuracy remained stable or slightly improved. Furthermore, AdaFocal-trained models showed a significant boost in OOD detection. By forcing the model to be "honest" about its uncertainty, it becomes much more effective at identifying samples that do not belong to the training distribution, outperforming previous state-of-the-art methods like FLSD-53.
4.3 Synergy with Post-hoc Methods
While AdaFocal produces inherently well-calibrated models, it remains compatible with post-training techniques like Temperature Scaling. When combined, these methods often achieve near-perfect calibration.
5. Discussion and Limitations
The success of AdaFocal suggests that calibration is a "moving target" during training. A single $\gamma$ value is a blunt instrument; high-confidence predictions may require high $\gamma$ to prevent arrogance, while low-confidence predictions may benefit from negative $\gamma$ to boost confidence.
However, the method does introduce some overhead. AdaFocal requires an additional forward pass on the validation set every epoch to update the bin statistics. While the authors demonstrate this adds only a small fraction to the total training time (e.g., 16 minutes added to a 7.7-hour run on CIFAR-10), it is a factor for consideration in resource-constrained environments. Additionally, while the method reduces the need for manual $\gamma$ tuning, it introduces new hyperparameters such as the number of bins and the update scaling factor ($\lambda$).
6. Conclusion
AdaFocal provides a robust framework for training "honestly confident" neural networks. By adaptively adjusting the loss function based on validation performance, it bridges the gap between accuracy and calibration. Future work may extend this approach to ensure "fair" calibration across different demographic groups, ensuring that model reliability is consistent across all segments of a population.
Summary updated