Abstract
While Mixup is a popular technique for improving deep neural network (DNN) accuracy, it does not consistently lead to better-calibrated models and can sometimes significantly increase overconfidence. This paper introduces Auto-Regularized-Confidence (ARC), a novel loss function that explicitly aligns a model’s confidence with its empirical accuracy. By acting as a regularizer alongside standard cross-entropy, ARC achieves state-of-the-art calibration without sacrificing predictive power, particularly on complex, fine-grained datasets where standard augmentation strategies often fail.
Introduction
Deep Neural Networks (DNNs) are notoriously overconfident. In a "well-calibrated" model, a confidence score of 90% should correspond to a 90% likelihood of being correct. However, modern DNNs frequently assign near-100% confidence even to incorrect predictions, a significant drawback for high-stakes applications like medical diagnosis or autonomous driving.
Data augmentation strategies like Mixup—which trains on linear interpolations of input pairs and their labels—are often assumed to improve calibration by "smoothing" decision boundaries. This research challenges that assumption, arguing that Mixup focuses on geometric smoothness rather than the actual uncertainty inherent in the data distribution. Consequently, while Mixup may improve accuracy, it can severely degrade calibration on complex, realistic datasets.
Methods: Auto-Regularized-Confidence (ARC)
The authors propose a training framework that augments standard Categorical Cross-Entropy (CE) with the Auto-Regularized-Confidence (ARC) loss. The ARC loss functions through a three-step process:
- Binning: Partitioning the model's confidence scores into $M$ discrete bins.
- Matching: Forcing the average confidence within each bin to match the actual accuracy of the samples in that bin.
- Variants: The authors developed $ARC_{V1}$ (matching average confidence to accuracy) and $ARC_{V2}$ (forcing individual samples to match the bin's accuracy).
The total loss is defined as $L(\theta) = CE + \beta \cdot ARC$, where $\beta$ is a hyperparameter controlling the calibration penalty. While CE acts as an "accelerator" for class discrimination, ARC serves as a "tether," pulling confidence back to levels justified by the model's actual performance.
Experimental Setup
The approach was evaluated on five datasets representing a range of complexity:
- CIFAR-10 & CIFAR-100: Standard small-image benchmarks.
- SVHN (Street View House Numbers): Digit recognition in natural scenes.
- Caltech Birds (CUB-200-2011) & Stanford Cars: Fine-grained classification tasks that represent more complex, real-world scenarios where simple linear interpolation (Mixup) may fail to capture underlying data geometry.
Results and Discussion
The most significant finding is that learning "smoother" decision thresholds is not sufficient for calibration. Calibration requires the model to understand the proportion of samples and class overlap, which Mixup does not explicitly model.
The Mixup Calibration Gap
On fine-grained datasets like Birds and Cars, Mixup was found to be detrimental to calibration. As shown in the experimental results, the Expected Calibration Error (ECE) for the Birds dataset jumped from ~2.4% in the baseline to over 14% when Mixup was applied.
ARC Performance
The combination of ARC + Mixup (A+M) consistently provided the best balance of accuracy and calibration. On CIFAR-10, the method achieved a remarkably low ECE of 1.03%, compared to 4.65% for Mixup alone. In the fine-grained tasks, ARC successfully mitigated the miscalibration introduced by Mixup, reducing ECE on the Birds dataset from 14.22% to 4.95% (in one configuration) or even lower in others, while often improving accuracy over the baseline.
ARC also outperformed other recent techniques like MMCE (Trainable Calibration Measures), which proved to be less stable in certain training scenarios.
Conclusion
ARC provides a robust mechanism for calibrating DNNs, especially when used in conjunction with Mixup. However, the method is sensitive to hyperparameters like $\beta$ and the bin count $M$. Additionally, because the ARC signal disappears if a model reaches 100% training accuracy, the authors recommend calculating the loss on a separate validation split. Future work may involve developing more sophisticated uncertainty quantifiers to drive the ARC loss beyond simple accuracy metrics.
Summary updated