Pristine Annotations-Based Multi-modal Trained Artificial Intelligence Solution to Triage Chest X-Ray for COVID-19

Tao Tan, Tao Tan 0002, Bipul Das, Ravi Soni, Mate Fejes, Sohan Ranjan, Daniel Attila Szabo, Vikram Melapudi, K. S. Shriram, Utkarsh Agrawal, László Ruskó, Zita Herczeg, Barbara Darázs, Pal Tegzes, Lehel Ferenczi, Rakesh Mullick, Gopal AvinashMICCAI 2021 · DOI

TL;DR

This paper introduces a novel AI system that leverages multi-modal learning, combining high-dimensional CT scans with lower-dimensional X-ray images, to improve the triage of COVID-19 patients. By training on both modalities and inferring solely on X-rays, the system achieves state-of-the-art classification and segmentation performance, addressing the critical need for rapid and accurate COVID-19 diagnosis.

What Problem Does This Paper Address?

The COVID-19 pandemic has placed an immense strain on healthcare systems, necessitating rapid and accurate diagnostic tools. While PCR tests are the gold standard, they are time-consuming and costly, limiting widespread use. Chest X-rays offer a faster, more accessible alternative, but their interpretation can be challenging for radiologists, especially in identifying subtle signs of COVID-19. Existing AI solutions for X-ray analysis often suffer from limited data, lack of high-quality annotations, and a potential domain shift when trained on one modality and applied to another. This paper tackles the challenge of improving AI-driven COVID-19 triage using X-rays by effectively bridging the gap between high-resolution CT data and readily available X-ray images, while also addressing the critical need for precise disease region segmentation.

Methods

The core innovation lies in a multi-modal training strategy that leverages both CT and X-ray data. The authors developed a pipeline that first generates synthetic X-ray (SXR) images from 3D CT volumes. This process involves simulating the X-ray beam attenuation through the CT data, effectively creating a 2D representation. Corresponding segmentation masks for disease regions are also generated from CT annotations.

For paired CT and X-ray data, the authors transfer the CT-derived disease masks to the synthetic X-rays and then to the real X-ray images. This "pristine annotation" process involves manual adjustment by trained staff to account for differences in image quality and patient positioning. The AI model, a deep learning convolutional neural network, is then trained using a combination of real X-ray images, synthetic X-rays, and these meticulously curated pristine annotations. The inference pipeline, however, exclusively uses real X-ray images, making it practical for clinical deployment.

Datasets

The study utilizes a combination of publicly available and in-house datasets. These include:

  • Real X-ray Images: Sourced from Kaggle Pneumonia RSNA, Kaggle Chest Dataset, PadChest Dataset, and IEEE github dataset, as well as in-house collections.
  • CT Volumes: Leveraged for generating synthetic X-rays and transferring annotations.
  • Synthetic X-ray Images (SXR): Generated from CT volumes.

The paper emphasizes the creation of "pristine mask annotations" (PMA) by refining transferred CT annotations with manual adjustments on real X-rays. This is a key differentiator from previous work that relied solely on X-ray annotations, which can be inconsistent.

Key Insights and Ideas

  • Multi-modal learning is crucial: Training on high-dimensional CT data, even indirectly through synthetic X-rays, significantly improves the performance of AI models inferring on lower-dimensional X-rays.
  • Synthetic data generation bridges the gap: Creating synthetic X-rays from CT allows for the augmentation of training data and the transfer of valuable annotation information from CT to X-ray.
  • Pristine annotations are key for segmentation: Manually refining transferred CT annotations leads to more accurate and reliable ground truth for disease segmentation, which is critical for severity assessment and monitoring.
  • Domain shift is a real challenge: The study highlights that while AI can learn from CT, direct inference on X-ray is essential for practical clinical application, and the multi-modal approach helps mitigate domain shift issues.

Key Results

  • Classification Performance: The multi-modal trained AI system achieved a classification AUC of 0.99 for COVID-19 vs. pneumonia+negative on the full testing dataset and 0.93 on the in-house subset.
  • Segmentation Performance: The Dice coefficient for segmenting COVID-19 disease regions improved significantly when using pristine annotations, reaching 0.62 compared to 0.57 when using X-ray direct annotations.
  • Annotation Consistency: The study demonstrated that pristine annotations (PMA) are more consistent with transferred CT annotations (TMA) than direct X-ray annotations (XMA), with a Dice overlap of 0.50 between XMA and PMA, and 0.47 between PMA and TMA.

Limitations and Future Work

  • Limitations:
    • The quality of synthetic X-rays, while useful, may not perfectly replicate real X-rays, and further investigation into generative networks for more realistic synthetic data is suggested.
  • There can be inconsistencies between X-ray and CT annotations due to registration errors and the dynamic nature of COVID-19 progression within the 48-hour window.
  • The study acknowledges that X-ray imaging inherently has limitations due to its 2D projection and potential for obscured information by overlying tissues.
  • Future Work:
    • Leveraging paired CT information to estimate disease severity and other higher-dimensional measures.
  • Exploring the direct deployment of AI on X-ray hardware with direct access to raw images to avoid vendor-specific post-processing.
  • Incorporating patient clinical parameters (e.g., blood oxygen levels, temperature) to further enhance diagnostic accuracy.

Summary updated

BibTeX
@inproceedings{tan2021pristine,
  author = {Tao Tan and Tao Tan and Bipul Das and Ravi Soni and Mate Fejes and Sohan Ranjan and Daniel Attila Szabo and Vikram Melapudi and K. S. Shriram and Utkarsh Agrawal and László Ruskó and Zita Herczeg and Barbara Darázs and Pal Tegzes and Lehel Ferenczi and Rakesh Mullick and Gopal Avinash},
  title = {{Pristine Annotations-Based Multi-modal Trained Artificial Intelligence Solution to Triage Chest X-Ray for COVID-19}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  doi = {10.1007/978-3-030-87234-2\_31},
}
Figure