Training Uncertainty-Aware Classifiers with Conformalized Deep Learning

Bat-Sheva Einbinder, Yaniv Romano, Matteo Sesia, Yanfei Zhou

Abstract: Deep neural networks are powerful tools to detect hidden patterns in data and leverage them to make predictions, but they are not designed to understand uncertainty and estimate reliable probabilities. In particular, they tend to be overconfident. We begin to address this problem in the context of multi-class classification by developing a novel training algorithm producing models with more dependable uncertainty estimates, without sacrificing predictive power. The idea is to mitigate overconfidence by minimizing a loss function, inspired by advances in conformal inference, that quantifies model uncertainty by carefully leveraging hold-out data. Experiments with synthetic and real data demonstrate this method can lead to smaller conformal prediction sets with higher conditional coverage, after exact calibration with hold-out data, compared to state-of-the-art alternatives.

Introduction

Deep neural networks (DNNs) have revolutionized predictive modeling but remain notorious for overconfidence. A standard classifier often assigns near-certainty to incorrect labels, particularly when encountering "aleatory uncertainty"—intrinsic noise such as blurry images or ambiguous medical data. While conformal inference is typically used as a post-processing step to generate prediction sets with statistical guarantees, it often suffers if the underlying model is biased. This research proposes integrating conformal inference directly into the training process to create "uncertainty-aware" classifiers.

Methodology

The core contribution is a novel uncertainty-aware loss function that optimizes for well-behaved uncertainty scores from the first epoch.

  1. The Loss Split: Training data is partitioned into two subsets. The first ensures predictive accuracy via standard cross-entropy loss, while the second evaluates a "conformal loss."
  2. Uniformity as a Signal: An ideal "oracle" model produces conformity scores that are uniformly distributed. The proposed loss penalizes deviations from this distribution using a differentiable version of the Kolmogorov-Smirnov test statistic.
  3. Differentiable Sorting: To handle the non-differentiable nature of ranking and sorting required for conformity scores, the authors employ "soft-sorting" algorithms, allowing the signal to propagate through backpropagation.

Experimental Results

The method was evaluated on synthetic data, CIFAR-10 (with added noise via RandomErasing), and a Credit Card Default dataset.

Quantitative Performance

As shown in the performance metrics, the proposed conformal training method significantly outperforms standard baselines in terms of both reliability and efficiency. Specifically, it achieves higher conditional coverage—meaning it is reliable for both "easy" and "hard" samples—while simultaneously producing smaller, more informative prediction sets.

Conditional coverage and average prediction set size as a function of training sample size. The proposed conformal method (blue) achieves higher conditional coverage and smaller set sizes compared to standard baselines.

Qualitative Analysis

The practical impact of this training is visible in image classification tasks. In instances of high ambiguity, such as corrupted images, standard models like Cross Entropy or Hybrid methods remain overconfident in a single (often wrong) label. In contrast, the Conformalized model correctly identifies the ambiguity, producing a prediction set that reflects the true uncertainty of the input.

Qualitative comparison of prediction sets on CIFAR-10. For the corrupted image (right), the conformal method provides a more balanced and realistic set of labels compared to overconfident or overly broad baselines.

Key Insights and Discussion

  • Conditional vs. Marginal Coverage: While standard methods guarantee 90% accuracy across a whole population, they often fail on difficult subgroups. This method pushes toward conditional coverage, aiming for 90% reliability for every type of input.
  • Robustness to Covariate Shift: The model proved more robust when test data was noisier than training data. On CIFAR-10, it maintained ~90% coverage for corrupted images, whereas the hybrid benchmark's coverage dropped to ~81%.
  • Limitations: The primary trade-off is computational overhead; training on CIFAR-10 took nearly double the time of standard methods. Additionally, the method requires a relatively large number of training samples to accurately estimate conformity score distributions.

Conclusion

By making deep learning models aware of the eventual conformal calibration step during training, they learn better internal representations of uncertainty. This synergy leads to classifiers that are not only accurate but also know when to be cautious, providing a significant step toward more reliable AI in high-stakes domains.

Summary updated

Abstract

Deep neural networks are powerful tools to detect hidden patterns in data and leverage them to make predictions, but they are not designed to understand uncertainty and estimate reliable probabilities. In particular, they tend to be overconfident. We begin to address this problem in the context of multi-class classification by developing a novel training algorithm producing models with more dependable uncertainty estimates, without sacrificing predictive power. The idea is to mitigate overconfidence by minimizing a loss function, inspired by advances in conformal inference, that quantifies model uncertainty by carefully leveraging hold-out data. Experiments with synthetic and real data demonstrate this method can lead to smaller conformal prediction sets with higher conditional coverage, after exact calibration with hold-out data, compared to state-of-the-art alternatives.
BibTeX
@inproceedings{einbinder2022training,
  author = {Bat-Sheva Einbinder and Yaniv Romano and Matteo Sesia and Yanfei Zhou},
  title = {{Training Uncertainty-Aware Classifiers with Conformalized Deep Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=mbUhZTUFCt},
}
Figure