Recalibration of deep learning models for abnormality detection in smartphone-captured chest radiograph

Po-Chih Kuo, Cheng Che Tsai, Diego M. López, Alexandros Karargyris, Tom J. Pollard, Alistair E. W. Johnson, Leo Anthony CeliNpj Digit. Medicine 2021 · OpenReview

TL;DR

This paper tackles the challenge of applying deep learning models trained on high-quality digital chest X-rays (CXRs) to lower-quality smartphone photographs of CXRs. They introduce a novel "recalibration" method that significantly restores performance on these photographic CXRs, making AI-assisted diagnostics more feasible in resource-limited settings.

What Problem Does This Paper Address?

Interpreting chest X-rays (CXRs) is crucial for diagnosing a wide range of pulmonary conditions. While deep learning models have shown remarkable accuracy in detecting abnormalities on high-resolution digital CXRs, their performance plummets when applied to images captured by smartphones. This is a significant hurdle because smartphone-based teleconsultation is becoming increasingly popular, especially in areas lacking immediate access to radiologists. Clinicians often resort to taking photos of CXR films with their phones to get quick "wet reads" from colleagues. The problem is that the "domain shift" – the difference in image quality, noise, and lighting between professional digital CXRs and smartphone photos – causes existing AI models to fail. This study aims to bridge this gap by developing a method to make these AI models robust to smartphone-captured CXRs.

Methods

The core of this paper is the recalibration method. Instead of training a new model from scratch on smartphone photos (which would require a massive dataset) or simply fine-tuning an existing model (which proved less effective), the authors developed a strategy to adapt pre-trained models.

Here's a breakdown of their approach:

  • Model-ORIG: This is the baseline model, trained on high-resolution digital CXRs from large databases like MIMIC-CXR.
  • Model-RECA (Recalibrated Model): This is their novel approach. They take the Model-ORIG and "recalibrate" it using augmented CXR images. These augmented images are created by applying various noise patterns, saturation changes, overexposure, contrast adjustments, motion blur, and other simulated photographic artifacts to original digital CXRs. The goal is to make the model learn to be robust to these common imperfections found in smartphone photos.
  • Model-TRNS (Transfer Learning Model): This model is created by taking Model-ORIG and fine-tuning it on a smaller dataset of actual smartphone-captured CXR photographs.
  • Model-PHOT (Photograph-based Model): This model is trained entirely from scratch on actual smartphone-captured CXR photographs.

They then evaluate these models against a "comparison reference" (performance on original digital CXRs) and against each other on various smartphone-captured CXR datasets.

Datasets

The study utilized two large, publicly available digital CXR databases:

  • MIMIC-CXR: A dataset of 64,588 patients from Beth Israel Deaconess Medical Center.
  • CheXpert: A dataset from Stanford Hospital with 65,240 patients.

From these, they created four derivative datasets of smartphone-captured CXR photographs for evaluation:

  • Photo-MMC: 1,759 photographs of MIMIC-CXR images.
  • Photo-CXP: 1,337 photographs of CheXpert images.
  • Photo-MED: 1,337 photographs taken by nine medical residents using their own smartphones, simulating an end-user scenario.
  • Photo-DEV: 2,020 photographs taken by a single physician using different devices and settings to test device-variance.

Key Insights and Ideas

  • The "Domain Shift" is Real and Damaging: The most striking insight is how drastically performance drops when AI models trained on pristine digital CXRs are applied to noisy smartphone photos. This highlights a critical bottleneck for deploying AI in real-world clinical settings where image quality is often suboptimal.
  • Recalibration Outperforms Simple Fine-Tuning: Their "recalibration" method, which uses augmented images to simulate photographic noise, proved more effective than simply fine-tuning a pre-trained model on a limited set of actual smartphone photos (Model-TRNS). This suggests that simulating a wide range of potential image degradations is a more robust strategy than relying on a small, specific set of real-world examples.
  • Smartphone Photos Can Be "Good Enough" with the Right Approach: The study demonstrates that with proper recalibration, AI models can achieve performance on smartphone photos that is comparable to their performance on original digital CXRs, making them viable for clinical assistance.
  • Generalizability is Key: The recalibrated model (Model-RECA) showed consistent performance across different datasets (MIMIC-CXR and CheXpert) and in simulated end-user scenarios, indicating its potential for broad applicability.
  • Explainability Matters: The use of Grad-CAM visualization showed that the recalibrated model was less distracted by noise (e.g., mistaking a clavicle for consolidation) compared to the original model, providing visual evidence of its improved robustness.

Key Results

The recalibrated model (Model-RECA) demonstrated significant performance recovery across various radiological findings:

  • Average AUROC Recovery: Model-RECA recovered 84.9% for cardiomegaly, 83.5% for edema, 53.2% for consolidation, 57.8% for atelectasis, 69.9% for pneumothorax, and 83.0% for pleural effusion, compared to the performance loss of the uncalibrated model.
  • AUROC Scores: For the six major findings, Model-RECA achieved AUROCs ranging from 0.75 to 0.88 on smartphone photographs, which was significantly higher than Model-ORIG (ranging from 0.61 to 0.77).
  • End-User Scenario: In a simulated scenario with medical residents taking photos, Model-RECA achieved an average AUROC of 0.75, matching the performance of the comparison reference and outperforming other models.
  • Device Variance: Model-RECA showed significantly higher reliability (measured by ICC) and a higher overall AUROC (0.80 ± 0.076) compared to Model-ORIG (0.74 ± 0.094) when tested across different devices and settings.

Limitations and Future Work

  • Dataset Origin: The digital CXR databases used for training were primarily from US populations. The authors acknowledge that recalibrating with photographs obtained from local populations in different regions would be ideal, as radiographic findings might differ.
  • Label Harmonization: The study notes that differences in how labels like "consolidation," "pneumonia," and "opacity" are defined across datasets could interfere with recalibration. Harmonizing these labels beforehand would improve consistency.
  • Comparison to Human Readers: The paper did not directly compare the recalibrated model's performance on smartphone photos to that of human radiologists interpreting the same photos.
  • Information Loss: The authors acknowledge that even with recalibration, a significant loss of information from high-resolution digital images to low-resolution smartphone photos cannot be fully compensated for.
  • Future Work: The authors suggest that their recalibration method is architecture-agnostic and could be applied to various deep learning models. They also propose that this approach can serve as an alternative to traditional transfer learning for addressing domain shift problems in medical imaging. The open-sourcing of their photograph datasets aims to encourage further research in this area.

Summary updated

Abstract

Image-based teleconsultation using smartphones has become increasingly popular. In parallel, deep learning algorithms have been developed to detect radiological findings in chest X-rays (CXRs). However, the feasibility of using smartphones to automate this process has yet to be evaluated. This study developed a recalibration method to build deep learning models to detect radiological findings on CXR photographs. Two publicly available databases (MIMIC-CXR and CheXpert) were used to build the models, and four derivative datasets containing 6453 CXR photographs were collected to evaluate model performance. After recalibration, the model achieved areas under the receiver operating characteristic curve of 0.80 (95% confidence interval: 0.78–0.82), 0.88 (0.86–0.90), 0.81 (0.79–0.84), 0.79 (0.77–0.81), 0.84 (0.80–0.88), and 0.90 (0.88–0.92), respectively, for detecting cardiomegaly, edema, consolidation, atelectasis, pneumothorax, and pleural effusion. The recalibration strategy, respectively, recovered 84.9%, 83.5%, 53.2%, 57.8%, 69.9%, and 83.0% of performance losses of the uncalibrated model. We conclude that the recalibration method can transfer models from digital CXRs to CXR photographs, which is expected to help physicians’ clinical works.
BibTeX
@inproceedings{kuo2021recalibration,
  author = {Po-Chih Kuo and Cheng Che Tsai and Diego M. López and Alexandros Karargyris and Tom J. Pollard and Alistair E. W. Johnson and Leo Anthony Celi},
  title = {{Recalibration of deep learning models for abnormality detection in smartphone-captured chest radiograph}},
  booktitle = {Npj Digit. Medicine},
  year = {2021},
  url = {https://openreview.net/forum?id=vRxJ5g0Yst},
}
Figure