TL;DR
This paper introduces a novel method for neural networks to learn their own confidence in predictions, enabling them to better detect when they encounter data outside their training distribution. This approach is simple, requires no extra labels, and outperforms existing methods for out-of-distribution detection, while also showing that misclassified in-distribution examples can serve as a proxy for out-of-distribution data during calibration.
What Problem Does This Paper Address?
Modern deep neural networks are incredibly powerful at making predictions on data similar to what they were trained on. However, they often fail spectacularly when presented with inputs that are significantly different from their training data. This is known as the "out-of-distribution" (OOD) detection problem. Worse still, these networks frequently provide highly confident predictions even for these out-of-distribution inputs, leading to a "silent failure" where the system appears to be working correctly but is actually making erroneous decisions. This lack of self-awareness is a major concern for AI safety, especially in critical applications. The paper aims to equip neural networks with a reliable way to express uncertainty about their predictions, thereby enabling them to flag potentially problematic inputs.
Methods
The core contribution is a new approach to train neural networks to output a "confidence estimate" alongside their class predictions. This is achieved by adding a parallel "confidence estimation branch" to the network, typically after the penultimate layer. This branch outputs a single scalar value between 0 and 1, representing the network's confidence in its prediction.
During training, the network is incentivized to produce high confidence for correct predictions and low confidence for incorrect ones. This is done by modifying the standard loss function. The network's predicted probabilities are "interpolated" between the original prediction and the true label, with the degree of interpolation controlled by the confidence score. If the confidence is high (close to 1), the prediction remains largely unchanged. If the confidence is low (close to 0), the prediction is pushed strongly towards the true label.
To prevent the network from simply outputting zero confidence for all inputs to minimize loss, a "confidence loss" term is added, which penalizes low confidence scores. A "budget parameter" ($\beta$) is introduced to control the trade-off between the task loss and the confidence loss, preventing the confidence estimates from becoming trivial.
The paper also proposes a technique to combat excessive regularization caused by the confidence branch, by applying the confidence-based prediction modification to only half of the batch at each iteration.
Datasets
The paper uses standard image classification datasets for training and evaluation:
- In-distribution datasets:
- SVHN (Street View Housing Numbers)
- CIFAR-10
- Out-of-distribution datasets:
- TinyImageNet (crop and resize variants)
- LSUN (crop and resize variants)
- iSUN
- Uniform Noise
- Gaussian Noise
- "All Images" (a combination of the above real image datasets)
The paper does not introduce new benchmark datasets but uses existing ones to compare its method against prior work.
Key Insights and Ideas
- Confidence as a direct signal for OOD detection: The paper demonstrates that a learned confidence score, rather than just the softmax output, is a more effective signal for distinguishing between in-distribution and out-of-distribution data.
- Misclassified in-distribution examples as OOD proxies: A crucial finding is that misclassified examples from a validation set can serve as a surprisingly effective substitute for actual out-of-distribution examples when calibrating the detection threshold. This significantly reduces the need for expensive and difficult-to-obtain OOD data.
- Simplicity and efficiency: The proposed confidence estimation branch is simple to implement, requires no additional labels, and adds minimal computational overhead during inference.
- Intuitive interpretability: The confidence scores provide an interpretable measure of the network's certainty, which can be valuable for human-in-the-loop systems.
Key Results
- The proposed confidence-based thresholding method consistently outperforms the baseline method (Hendrycks & Gimpel, 2017) and ODIN (Liang et al., 2018) across various network architectures and OOD datasets, particularly in terms of detection error and AUROC/AUPR metrics.
- When using misclassified in-distribution examples as a proxy for OOD data, the detection performance is only slightly reduced compared to using true OOD examples, making calibration much more practical.
- The confidence estimation training does not significantly degrade the classification accuracy of the models.
Limitations and Future Work
- Selecting the detection threshold ($\delta$): While the paper addresses this by proposing the use of misclassified in-distribution examples, selecting the optimal threshold can still be challenging and may require careful tuning.
- Generalization to other tasks: The current work focuses on image classification. The authors suggest future work to apply learned confidence estimates to other domains like semantic segmentation and natural language understanding.
- More complex "hint" schemes: The paper mentions exploring more human-inspired schemes for giving "hints" to the network, implying a richer interaction beyond simple confidence adjustments.
Summary updated