CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn L. Ball, Katie S. Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, Andrew Y. Ng

TL;DR

This paper introduces CheXpert, a massive dataset of over 224,000 chest X-rays with labels for 14 common observations, crucially including uncertainty. They developed an automated labeling system that captures these uncertainties from radiology reports and explored various methods for training deep learning models to leverage this uncertainty information, achieving performance competitive with, and in some cases exceeding, radiologists.

What Problem Does This Paper Address?

Automated interpretation of medical images, particularly chest X-rays, holds immense potential for improving healthcare by assisting clinicians, prioritizing workflows, and enabling large-scale screening. However, progress in this area has been hampered by a lack of large, well-annotated datasets with robust ground truth and expert performance benchmarks. Existing datasets often rely on automated labeling from reports, which can be noisy, or lack sufficient scale and expert-level validation. A key challenge in medical imaging is the inherent uncertainty in diagnoses, which is often reflected in radiology reports but not adequately captured in traditional binary labels. This paper addresses the need for a large-scale, expertly validated dataset that explicitly accounts for uncertainty in radiograph interpretation.

Methods

The core of the paper involves two main components: an automated labeling system and novel approaches to handling uncertainty in model training.

  • Automated Labeling System: The authors developed a rule-based labeler to extract 14 common observations from free-text radiology reports. This system goes beyond simple keyword matching by employing mention extraction, classification (identifying positive, negative, and uncertain mentions), and aggregation. Crucially, it captures uncertainty by identifying phrases that indicate doubt, ambiguity, or a lack of definitive findings.
  • Uncertainty Handling in Model Training: Recognizing that uncertainty is prevalent and important, the paper explores several strategies for incorporating these "u" labels into deep learning models:
  • U-Ignore: Simply discarding uncertain labels, serving as a baseline.
  • U-Zeros/U-Ones: Mapping uncertain labels to either negative (0) or positive (1), exploring simple binary interpretations.
  • U-SelfTrained: A semi-supervised approach where an initial model trained on certain labels is used to predict labels for uncertain cases, which are then incorporated into further training.
  • U-MultiClass: Treating uncertainty as a distinct third class, allowing the model to learn to predict probabilities for positive, negative, and uncertain outcomes.

Datasets

  • CheXpert Dataset: The primary contribution is the CheXpert dataset, comprising 224,316 chest radiographs from 65,240 patients. This dataset is labeled for 14 common chest radiographic observations.
  • Validation Set: A subset of 200 studies was manually annotated by 3 board-certified radiologists to evaluate different uncertainty approaches.
  • Test Set: Another set of 500 studies was annotated by a consensus of 5 board-certified radiologists to serve as a strong ground truth for comparing model performance against radiologists.
  • Report Evaluation Set: 1000 reports were used to evaluate the performance of their automated labeler against a previous method.

Key Insights and Ideas

  • Uncertainty is a First-Class Citizen: The paper strongly advocates for treating uncertainty not as noise to be discarded, but as a critical signal that can improve model performance.
  • Context Matters for Uncertainty: Different uncertainty approaches are more effective for different pathologies. For instance, treating uncertain findings as positive (U-Ones) worked well for Atelectasis and Edema, suggesting these "uncertain" mentions often indicate likely findings.
  • The "Borderline Case" Problem: Explicitly modeling uncertainty (U-MultiClass) is crucial for handling ambiguous cases, such as "minimal cardiac enlargement," which are difficult to definitively label as positive or negative. Ignoring these cases can lead to poor performance.
  • Automated Labeling Can Be Powerful, But Needs Nuance: Their automated labeler, which captures uncertainty, performs significantly better than previous methods, demonstrating the value of nuanced report parsing.
  • Deep Learning Models Can Approach Radiologist Performance: The study shows that well-trained models, especially those leveraging uncertainty, can achieve performance comparable to or exceeding that of individual radiologists on specific tasks.

Key Results

  • The CheXpert dataset is one of the largest publicly available chest X-ray datasets.
  • Their automated labeler achieved higher F1 scores across mention extraction, negation detection, and uncertainty detection compared to a previous NIH labeler.
  • On the validation set, the U-MultiClass approach showed significant improvements for Cardiomegaly (AUC=0.854 vs. U-Ignore AUC=0.828), and U-Ones for Atelectasis (AUC=0.858 vs. U-Zeros AUC=0.811).
  • On the test set, the final model achieved AUCs of at least 0.9 for most competition tasks, with the best performance on Pleural Effusion (0.97) and worst on Atelectasis (0.85).
  • For Cardiomegaly, Edema, and Pleural Effusion, the model's ROC and PR curves consistently lay above the operating points of all three benchmark radiologists.

Limitations and Future Work

  • Limited Patient History: The study acknowledges that neither the model nor the radiologists had access to patient history or previous examinations, which can significantly impact diagnostic performance.
  • Statistical Significance: No formal statistical tests were performed to definitively prove the model's superiority over radiologists, only visual comparison of operating points.
  • Future Work: The authors hope the dataset will spur further development in chest radiograph interpretation models. They also suggest exploring more sophisticated uncertainty handling techniques and integrating patient history for improved diagnostic accuracy. The visualization of model attention using Grad-CAMs suggests potential for interpretability, which could be further explored.

Summary updated

Abstract

Large, labeled datasets have driven deep learning methods to achieve expert-level performance on a variety of medical imaging tasks. We present CheXpert, a large dataset that contains 224,316 chest radiographs of 65,240 patients. We design a labeler to automatically detect the presence of 14 observations in radiology reports, capturing uncertainties inherent in radiograph interpretation. We investigate different approaches to using the uncertainty labels for training convolutional neural networks that output the probability of these observations given the available frontal and lateral radiographs. On a validation set of 200 chest radiographic studies which were manually annotated by 3 board-certified radiologists, we find that different uncertainty approaches are useful for different pathologies. We then evaluate our best model on a test set composed of 500 chest radiographic studies annotated by a consensus of 5 board-certified radiologists, and compare the performance of our model to that of 3 additional radiologists in the detection of 5 selected pathologies. On Cardiomegaly, Edema, and Pleural Effusion, the model ROC and PR curves lie above all 3 radiologist operating points. We release the dataset to the public as a standard benchmark to evaluate performance of chest radiograph interpretation models. The dataset is freely available at https://stanfordmlgroup.github.io/competitions/chexpert .
BibTeX
@inproceedings{irvin2019chexpert,
  author = {Jeremy Irvin and Pranav Rajpurkar and Michael Ko and Yifan Yu and Silviana Ciurea-Ilcus and Chris Chute and Henrik Marklund and Behzad Haghgoo and Robyn L. Ball and Katie S. Shpanskaya and Jayne Seekins and David A. Mong and Safwan S. Halabi and Jesse K. Sandberg and Ricky Jones and David B. Larson and Curtis P. Langlotz and Bhavik N. Patel and Matthew P. Lungren and Andrew Y. Ng},
  title = {{CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.3301590},
}
Figure