Improving Uncertainty Estimation with Confidence-Aware Training Data

Alexey Zaytsev, Sergey Korchagin, Ekaterina Zaychenkova, Aleksei Khalin, Aleksandr Yugay, Alexey Zaytsev 0002, Egor I. ErshovWACV 2025 · DOI

Introduction

In high-stakes domains such as medical diagnostics, the reliability of AI predictions is as critical as their accuracy. Current deep learning models frequently suffer from overconfidence, primarily because they are trained on "hard" binary labels (e.g., 0 or 1). This approach forces models to make definitive predictions even when the underlying data is inherently ambiguous—a phenomenon known as aleatoric uncertainty. While epistemic uncertainty (uncertainty due to lack of data) can be reduced with more training examples, aleatoric uncertainty is irreducible and stems from noise or ambiguity in the data itself. This paper introduces a novel method to capture both types of uncertainty by incorporating expert confidence scores, or "soft labels," into the training process.

Methods: Confidence-Aware Ensembles

The authors propose a Confidence-Aware Ensemble (CAE) approach designed to decompose total predictive uncertainty into its epistemic and aleatoric components using the Law of Total Variance.

The Confidence-Aware Ensemble (CAE) pipeline, illustrating the two-stage training process using hard labels and expert-provided soft labels.

The pipeline consists of two distinct stages:

  1. Epistemic Estimation: An ensemble of $K$ models is trained on standard hard labels. The variance between these models' predictions serves as a proxy for epistemic uncertainty, representing the disagreement between different hypotheses.
  2. Aleatoric Estimation: The ensemble is then fine-tuned to predict soft labels—probabilities or confidence scores provided by human experts. The aleatoric uncertainty is calculated as the expected variance of these predicted probabilities.

By combining these components, the CAE produces a "Total Variance" score that more accurately reflects the likelihood of misclassification, particularly in cases of high data ambiguity. This differs from standard approaches that attempt to derive both types of uncertainty from hard labels alone, which often leads to biased and overconfident results.

Datasets

The methodology was validated across three diverse datasets:

  • Synthetic Data: High-dimensional simulations used to establish the theoretical benefits of soft labels for model calibration.
  • BloodyWell: A blood-typing dataset containing 3,139 images. Each image is annotated with both hard labels from medical records and soft labels representing the consensus and confidence of six serology experts.
  • LIDC-IDRI: A public dataset of 1,018 lung CT scans. Radiologist annotations were used to generate soft segmentation maps, capturing pixel-level expert uncertainty.

Key Results and Discussion

The CAE method demonstrated a significant leap in the quality of uncertainty estimation across all benchmarks, particularly in its ability to identify potential errors.

Accuracy vs. rejection curves and uncertainty distribution. The proposed method (Ours) significantly improves error detection compared to the baseline.

Performance Gains

  • Error Detection: On the BloodyWell classification task, the CAE improved the Area Above the rejection Curve (AAC) by a factor of two compared to standard ensemble baselines. As shown in the rejection curves, the model's ability to identify and "reject" potentially incorrect samples for human review is substantially more efficient.
  • Segmentation: In medical image segmentation (histology and lung nodules), the approach yielded a 62% improvement in uncertainty quality.
  • The Synergy Effect: A critical finding is that a small number of soft labels can have a disproportionately large impact. In synthetic tests, adding just 125 soft labels to a dataset of 2,000 hard labels reduced the Mean Absolute Error (MAE) by 28%, suggesting that expert confidence is a highly dense information signal.

Conclusion and Limitations

By embracing expert disagreement rather than treating it as noise, the CAE framework provides a more nuanced and reliable measure of model uncertainty. The primary limitation remains the cost of expertise; obtaining high-quality soft labels requires multiple experts to evaluate the same data. Future work will focus on generating these labels more efficiently and developing methods that are more robust to "noisy" expert confidence scores where agreement is low.

Summary updated

BibTeX
@inproceedings{zaytsev2025improving,
  author = {Alexey Zaytsev and Sergey Korchagin and Ekaterina Zaychenkova and Aleksei Khalin and Aleksandr Yugay and Alexey Zaytsev and Egor I. Ershov},
  title = {{Improving Uncertainty Estimation with Confidence-Aware Training Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00776},
}
Figure