Confidence Estimation Using Unlabeled Data

Chen Li, Xiaoling Hu, Chao Chen

Abstract

Deep neural networks (DNNs) are frequently overconfident, often assigning high probability to incorrect predictions. While confidence estimation is crucial for safety-critical applications, existing methods typically rely on fully labeled datasets. This paper introduces the first confidence estimation method designed for semi-supervised settings. By utilizing "training consistency"—the stability of model predictions across epochs—as a surrogate for confidence, we propose a consistency ranking loss. Our approach achieves state-of-the-art performance in image classification and segmentation tasks and demonstrates significant utility in downstream active learning.

Introduction

In domains such as autonomous driving and medical diagnostics, the reliability of a model's prediction is as vital as its accuracy. However, modern DNNs are notoriously poorly calibrated, exhibiting "overconfidence" where the softmax output does not accurately reflect the true likelihood of correctness.

While various techniques have been developed to calibrate these scores, they generally require large amounts of labeled data. In practice, labels are often scarce and expensive to obtain. This research addresses the challenge of estimating confidence when only a small fraction of the data is labeled, leveraging the vast pool of unlabeled data to improve model reliability.

Methodology: Training Consistency as a Proxy

The core innovation of this work is the use of Training Consistency as a surrogate for confidence.

1. The Consistency Metric

The authors observe that samples located far from the decision boundary tend to receive stable predictions throughout the training process. Conversely, samples near the boundary—often the most ambiguous ones—cause the model's predictions to fluctuate between classes across sequential epochs. By tracking how often a model’s prediction for a specific sample (labeled or unlabeled) changes, we can derive a measure of the model's certainty without needing a ground-truth label.

2. Consistency Ranking Loss ($L_{cons}$)

Rather than attempting to regress a specific probability value, the proposed method employs a pairwise ranking loss. The objective is to ensure the model ranks its outputs according to their observed consistency. If Sample A demonstrates higher consistency than Sample B during training, the model is penalized if its final confidence score for A is not higher than for B.

3. Hybrid Loss Function

The total training objective is a multi-task loss that combines:

  • Cross-Entropy Loss: Ensures prediction accuracy on labeled data.
  • Correctness Ranking Loss: Calibrates confidence based on ground-truth labels.
  • Consistency Ranking Loss ($L_{cons}$): Leverages both labeled and unlabeled data to refine confidence based on prediction stability.

Experimental Evaluation

The method was validated across three distinct domains to ensure robustness:

  • Image Classification: Evaluated on CIFAR-10 and CIFAR-100 datasets.
  • Medical Image Segmentation: Tested on the ISIC2017 skin lesion dataset, requiring the method to produce dense, pixel-level confidence maps.
  • Clinical Survival Analysis: Applied to a Cancer Survival dataset to predict 5-year outcomes from histopathological features.

Key Results and Insights

The proposed framework achieved state-of-the-art results in confidence estimation, particularly in data-constrained environments.

  • Improved Ranking Performance: On CIFAR-100 with only 2,500 labels, the method significantly outperformed standard Softmax and Monte Carlo Dropout in terms of the Area Under Risk-Coverage curve (AURC).
  • Active Learning Efficiency: By using the estimated confidence to identify the most informative unlabeled samples for human annotation, the method accelerated accuracy gains compared to traditional active learning baselines.
  • Reliable Segmentation: In medical tasks, the model produced "honest" confidence maps, correctly identifying lesion boundaries as areas of high uncertainty where errors were most likely to occur.

Discussion and Limitations

The central insight of this research is that unlabeled data provides a powerful signal for reliability. While most semi-supervised learning focuses on improving accuracy, this work demonstrates that unlabeled data is equally valuable for uncertainty estimation.

One limitation is the memory overhead required to track prediction histories across epochs. Additionally, while the method excels at in-distribution uncertainty, its efficacy in out-of-distribution (OOD) detection remains a subject for future study. Integrating this consistency ranking into more advanced semi-supervised frameworks like FixMatch or Mean Teacher represents a promising direction for future work.

Conclusion

This paper presents a novel approach to confidence estimation that thrives in the absence of abundant labels. By transforming training dynamics into a ranking signal, the proposed method bridges the gap between semi-supervised learning and model reliability, providing a robust tool for deploying deep learning in high-stakes environments.

Summary updated

Abstract

Overconfidence is a common issue for deep neural networks, limiting their deployment in real-world applications. To better estimate confidence, existing methods mostly focus on fully-supervised scenarios and rely on training labels. In this paper, we propose the first confidence estimation method for a semi-supervised setting, when most training labels are unavailable. We stipulate that even with limited training labels, we can still reasonably approximate the confidence of model on unlabeled samples by inspecting the prediction consistency through the training process. We use training consistency as a surrogate function and propose a consistency ranking loss for confidence estimation. On both image classification and segmentation tasks, our method achieves state-of-the-art performances in confidence estimation. Furthermore, we show the benefit of the proposed method through a downstream active learning task. The code is available at https://github.com/TopoXLab/consistency-ranking-loss
BibTeX
@inproceedings{li2023confidence,
  author = {Chen Li and Xiaoling Hu and Chao Chen},
  title = {{Confidence Estimation Using Unlabeled Data}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=sOXU-PEJSgQ},
}
Figure