AdaFocal: Calibration-aware Adaptive Focal Loss

Arindam Ghosh, Thomas Schaaf, Matthew R. Gormley

Abstract

Modern neural networks often suffer from miscalibration, exhibiting overconfidence in their predictions. While training with Focal Loss—controlled by a hyperparameter $\gamma$—has been shown to improve calibration compared to standard cross-entropy, existing approaches typically use a fixed or heuristic-based $\gamma$. We propose AdaFocal, a calibration-aware adaptive focal loss that dynamically modifies $\gamma$ for different groups of samples based on the model's performance on a validation set. By utilizing both focal and inverse-focal loss, AdaFocal addresses both under- and over-confidence. Our evaluations across various image recognition and NLP tasks demonstrate that AdaFocal significantly improves calibration and out-of-distribution detection while maintaining competitive accuracy.

1. Introduction

In many real-world applications, from autonomous driving to medical diagnosis, it is crucial that a neural network's confidence scores reflect the actual probability of being correct. This property is known as calibration. Unfortunately, modern deep neural networks are notoriously poorly calibrated, often producing highly overconfident predictions even when they are wrong.

Recent research highlighted that the Focal Loss, originally designed to handle class imbalance in object detection, acts as a regularizer that improves calibration by penalizing overconfident predictions. This effect is governed by the parameter $\gamma$. However, a fixed $\gamma$ is often insufficient, as different samples may require different levels of regularization. While sample-dependent heuristics like FLSD-53 have been proposed, they often lack generalization. AdaFocal addresses this by introducing a principled, adaptive approach to parameter selection.

2. The AdaFocal Approach

AdaFocal (Calibration-aware Adaptive Focal Loss) treats $\gamma$ as a dynamic parameter that evolves during training based on real-time feedback from the model's performance.

2.1 Validation-Driven Feedback and Binning

To determine how to adjust $\gamma$, AdaFocal monitors the model's calibration on a validation set. The probability space is divided into several "confidence bins" (e.g., 15 bins). For each bin, the algorithm calculates whether the model is currently overconfident (accuracy < confidence) or under-confident (accuracy > confidence).

2.2 Adaptive Update Rule

Instead of a global setting, AdaFocal maintains specific $\gamma$ values for different groups of samples. The update rule for $\gamma_t$ at step $t$ depends on the value from the previous step $\gamma_{t-1}$ and the observed calibration error in the corresponding bin. If a bin shows overconfidence, $\gamma$ is increased to increase regularization; if it shows under-confidence, $\gamma$ is decreased.

2.3 Inverse-Focal Loss

A key innovation in AdaFocal is the use of Inverse-Focal Loss. While standard Focal Loss (positive $\gamma$) reduces the weight of easy, high-confidence samples to curb overconfidence, AdaFocal allows $\gamma$ to become negative. This "inverse" application pushes the scores of under-confident samples higher, ensuring the model is not excessively "shy" about its correct predictions.

3. Experimental Evaluation

The researchers evaluated AdaFocal across diverse benchmarks to ensure its robustness:

  • Computer Vision: Experiments were conducted on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet using architectures like ResNet, WideResNet, and DenseNet.
  • Natural Language Processing: The method was tested on the 20 Newsgroups dataset using both CNNs and fine-tuned BERT models.
  • Out-of-Distribution (OOD) Detection: The models were tested on SVHN and corrupted datasets (CIFAR-10-C) to evaluate their ability to identify unfamiliar data.

4. Key Results

4.1 Superior Calibration

AdaFocal consistently achieved the lowest Expected Calibration Error (ECE) across nearly all tested configurations. For instance, on CIFAR-100 with a ResNet-50 architecture, AdaFocal reduced the ECE from 17.17% (cross-entropy) to a mere 1.36%.

4.2 Accuracy and OOD Performance

Crucially, AdaFocal improves calibration without sacrificing predictive performance. In many cases, top-1 accuracy remained stable or slightly improved. Furthermore, AdaFocal-trained models showed a significant boost in OOD detection. By forcing the model to be "honest" about its uncertainty, it becomes much more effective at identifying samples that do not belong to the training distribution, outperforming previous state-of-the-art methods like FLSD-53.

4.3 Synergy with Post-hoc Methods

While AdaFocal produces inherently well-calibrated models, it remains compatible with post-training techniques like Temperature Scaling. When combined, these methods often achieve near-perfect calibration.

5. Discussion and Limitations

The success of AdaFocal suggests that calibration is a "moving target" during training. A single $\gamma$ value is a blunt instrument; high-confidence predictions may require high $\gamma$ to prevent arrogance, while low-confidence predictions may benefit from negative $\gamma$ to boost confidence.

However, the method does introduce some overhead. AdaFocal requires an additional forward pass on the validation set every epoch to update the bin statistics. While the authors demonstrate this adds only a small fraction to the total training time (e.g., 16 minutes added to a 7.7-hour run on CIFAR-10), it is a factor for consideration in resource-constrained environments. Additionally, while the method reduces the need for manual $\gamma$ tuning, it introduces new hyperparameters such as the number of bins and the update scaling factor ($\lambda$).

6. Conclusion

AdaFocal provides a robust framework for training "honestly confident" neural networks. By adaptively adjusting the loss function based on validation performance, it bridges the gap between accuracy and calibration. Future work may extend this approach to ensure "fair" calibration across different demographic groups, ensuring that model reliability is consistent across all segments of a population.

Summary updated

Abstract

Much recent work has been devoted to the problem of ensuring that a neural network's confidence scores match the true probability of being correct, i.e. the calibration problem. Of note, it was found that training with focal loss leads to better calibration than cross-entropy while achieving similar level of accuracy \cite{mukhoti2020}. This success stems from focal loss regularizing the entropy of the model's prediction (controlled by the parameter $\gamma$), thereby reining in the model's overconfidence. Further improvement is expected if $\gamma$ is selected independently for each training sample (Sample-Dependent Focal Loss (FLSD-53) \cite{mukhoti2020}). However, FLSD-53 is based on heuristics and does not generalize well. In this paper, we propose a calibration-aware adaptive focal loss called AdaFocal that utilizes the calibration properties of focal (and inverse-focal) loss and adaptively modifies $\gamma_t$ for different groups of samples based on $\gamma_{t-1}$ from the previous step and the knowledge of model's under/over-confidence on the validation set. We evaluate AdaFocal on various image recognition and one NLP task, covering a wide variety of network architectures, to confirm the improvement in calibration while achieving similar levels of accuracy. Additionally, we show that models trained with AdaFocal achieve a significant boost in out-of-distribution detection.
BibTeX
@inproceedings{ghosh2022adafocal,
  author = {Arindam Ghosh and Thomas Schaaf and Matthew R. Gormley},
  title = {{AdaFocal: Calibration-aware Adaptive Focal Loss}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=kUOm0Fdtvh},
}
Figure