Quantifying and Leveraging Classification Uncertainty for Chest Radiograph Assessment

Florin C. Ghesu, Bogdan Georgescu, Eli Gibson, Sebastian Gündel, Mannudeep K. Kalra, Ramandeep Singh, Subba R. Digumarthy, Sasa Grbic, Dorin Comaniciu

Introduction

Interpreting chest X-rays (CXR) is a complex task characterized by significant inter-rater variability and inherent ambiguity. Standard deep learning models typically employ softmax classifiers that produce overconfident point estimates, failing to distinguish between aleatoric uncertainty (data noise) and epistemic uncertainty (model ignorance). This research introduces a framework that explicitly quantifies uncertainty using Subjective Logic to improve diagnostic reliability and model robustness.

Methodology

The proposed approach moves beyond standard probabilistic outputs by utilizing Subjective Logic and the Dempster-Shafer theory of evidence. Instead of a single probability, the model outputs evidence values ($e^+$ and $e^-$) that parameterize a Beta distribution.

Beta distributions representing different states of model confidence and uncertainty: (a) Confident negative, (b) Confident positive, and (c) High uncertainty.

Uncertainty ($\hat{u}$) is derived directly from the total evidence. When evidence is scarce, $\hat{u}$ approaches 1. The model is trained by optimizing the Bayes risk with a Beta-distributed prior, incorporating a Kullback-Leibler (KL) divergence term to regularize the model toward high uncertainty in the absence of conclusive evidence. This framework treats uncertainty as an orthogonal measure to the prediction, distinguishing between balanced evidence (ambiguity) and a lack of evidence (ignorance).

Experimental Setup

The framework was evaluated on two large-scale datasets: ChestX-Ray8 (112,120 images) and PLCO (185,421 images). To establish a ground truth, expert consensus sets were created where multiple board-certified radiologists reviewed test images to resolve label noise inherent in NLP-extracted reports.

Results

A key application of this framework is sample rejection. By identifying and "rejecting" the most uncertain cases for manual human review, the system's performance on the remaining automated cases improves significantly.

F1-score improvement as a function of the rejected data fraction for Granuloma and Fibrosis findings.

On the PLCO dataset, ROC-AUC increased from 0.86 to 0.91 by rejecting the 25% most uncertain cases. Furthermore, the model's uncertainty was found to be a strong indicator of label noise; uncertainty was significantly higher (0.41 vs 0.16) in cases where experts disagreed with the original NLP-derived labels.

Uncertainty-driven Bootstrapping

The authors also demonstrate that uncertainty can be used to clean training data. By filtering out the top 15% most uncertain training samples—often those with noisy or incorrect labels—the model's performance on clean test sets improved. For instance, Pleural Effusion AUC rose from 0.89 to 0.93 through this bootstrapping process. This strategy allows the model to become more robust against the noisy labels common in large medical datasets.

Conclusion

This work demonstrates that quantifying uncertainty is not just a safety feature but a tool for performance enhancement. By leveraging Subjective Logic, AI systems can identify their own limitations, effectively manage label noise, and optimize the human-AI diagnostic workflow. Future work may involve extending this to multi-label frameworks and exploring robust M-estimators to further refine the handling of uncertain samples.

Summary updated

Abstract

The interpretation of chest radiographs is an essential task for the detection of thoracic diseases and abnormalities. However, it is a challenging problem with high inter-rater variability and inherent ambiguity due to inconclusive evidence in the data, limited data quality or subjective definitions of disease appearance. Current deep learning solutions for chest radiograph abnormality classification are typically limited to providing probabilistic predictions, relying on the capacity of learning models to adapt to the high degree of label noise and become robust to the enumerated causal factors. In practice, however, this leads to overconfident systems with poor generalization on unseen data. To account for this, we propose an automatic system that learns not only the probabilistic estimate on the presence of an abnormality, but also an explicit uncertainty measure which captures the confidence of the system in the predicted output. We argue that explicitly learning the classification uncertainty as an additional measure to the predicted output, is essential to account for the inherent variability characteristic of this data. Experiments were conducted on two datasets of chest radiographs of over 85,000 patients. Sample rejection based on the predicted uncertainty can significantly improve the ROC-AUC, e.g., by 8% to 0.91 with an expected rejection rate of under 25%. Eliminating training samples using uncertainty-driven bootstrapping, enables a significant increase in robustness and accuracy. In addition, we present a multi-reader study showing that the predictive uncertainty is indicative of reader errors.
BibTeX
@misc{ghesu2019quantifying,
  author = {Florin C. Ghesu and Bogdan Georgescu and Eli Gibson and Sebastian Gündel and Mannudeep K. Kalra and Ramandeep Singh and Subba R. Digumarthy and Sasa Grbic and Dorin Comaniciu},
  title = {{Quantifying and Leveraging Classification Uncertainty for Chest Radiograph Assessment}},
  year = {2019},
  eprint = {1906.07775},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.07775},
}
Figure