Prospective Multi-Site Validation of AI to Detect Tuberculosis and Chest X-Ray Abnormalities

Sahar Kazemzadeh, Atilla P. Kiraly, Zaid Nabulsi, Nsala Sanjase, Minyoi M. Maimbolwa, Brian Shuma, Shahar Jamshy, Christina Chen, Arnav Agharwal, Charles T. Lau, Andrew Sellergren, D. I. Golden, Jin Yu, E H Wu, Yossi Matias, Katherine Chou, Greg S. Corrado, Shravya Shetty, Daniel Tse, Krishnan Eswaran, Yun Liu, Rory Pilgrim, Monde Muyoyeta, Shruthi PrabhakaraNEJM 2024 · DOI

TL;DR

This study prospectively validates two AI systems for interpreting chest X-rays (CXRs) to detect tuberculosis (TB) and other abnormalities in a high-burden setting in Zambia. The TB-detecting AI demonstrated comparable performance to radiologists, while the abnormality-detecting AI met its performance targets, suggesting AI's potential to improve TB screening accessibility and efficiency, especially in resource-constrained areas.

What Problem Does This Paper Address?

Tuberculosis (TB) remains a significant global health challenge, particularly in low- and middle-income countries (LMICs). A major hurdle in combating TB is the limited access to and high cost of accurate diagnostic tools, leading to delayed diagnosis and increased transmission. Chest X-rays (CXRs) are a widely available screening tool, but their interpretation requires trained radiologists, who are scarce in many LMICs. This scarcity, coupled with significant variability in human interpretation, hinders effective TB screening. While artificial intelligence (AI) shows promise for automating CXR analysis, prospective validation in real-world, high-burden settings, especially those with a high prevalence of HIV (which can alter TB presentation), is crucial. This paper addresses the need for such validation by evaluating two AI systems designed to detect TB and general CXR abnormalities in a Zambian population with a high TB and HIV burden.

Methods

The researchers conducted a prospective, multi-site study in Zambia involving three health facilities. They employed two cloud-based AI systems: one for TB detection (TB AI) and another for detecting general CXR abnormalities (abnormality AI). Participants were adults with TB symptoms, close contacts of TB patients, or newly diagnosed with HIV. Digital CXRs were de-identified and uploaded for AI analysis. The TB AI's performance was evaluated against two thresholds: a high-sensitivity threshold and a balanced threshold designed to mimic radiologist performance. For comparison, 10 radiologists independently reviewed the CXRs for TB. The abnormality AI was evaluated against a high-sensitivity target suitable for triaging. The reference standard for TB was microbiological testing (Xpert Ultra, smear, and culture), while the reference standard for CXR abnormalities was the majority vote of three experienced radiologists. Statistical analyses focused on comparing AI performance to radiologists and WHO targets using noninferiority margins.

Datasets

The study utilized data from 1932 participants enrolled across three health facilities in Lusaka, Zambia. Of these, 1910 had evaluable CXRs. The final TB AI analysis included 1827 patients, while the abnormality AI analysis included 1910 patients. The study did not introduce a new benchmark dataset but rather used real-world clinical data from a specific population to validate existing AI models.

Key Insights and Ideas

  • AI's potential in high-burden settings: The study demonstrates that AI can perform comparably to human radiologists in detecting TB, even in a complex population with a high TB and HIV burden. This is a critical step towards making advanced diagnostic capabilities more accessible.
  • HIV's impact on TB presentation: The findings highlight how HIV significantly impacts TB presentation on CXRs, leading to lower sensitivity for both AI and radiologists in HIV-positive individuals. This underscores the need for AI models that can adapt to these variations.
  • Dual-purpose AI: The development and evaluation of an abnormality-detecting AI alongside the TB AI suggest a future where a single CXR interpretation can simultaneously screen for TB and identify other potentially actionable findings, offering a more comprehensive diagnostic approach.
  • Threshold tuning is key: The study emphasizes that AI performance is highly dependent on the chosen thresholds. Different thresholds can prioritize sensitivity or specificity, allowing for customization based on specific clinical needs and resource constraints.

Key Results

  • TB AI Performance:
    • At a high-sensitivity threshold, the TB AI achieved 87% sensitivity and 70% specificity, outperforming radiologists in sensitivity (11% higher) but not specificity.
  • At a balanced threshold, the TB AI's sensitivity (78%) and specificity (82%) were comparable to radiologists' mean sensitivity (76%) and specificity (82%).
  • Neither the TB AI nor the radiologists consistently met the WHO targets of 90% sensitivity and 70% specificity, particularly in the HIV-positive subgroup.
  • Abnormality AI Performance:
    • The abnormality AI achieved a sensitivity of 97% and a specificity of 79% at its high-sensitivity threshold, meeting the prespecified targets of 90% sensitivity and 50% specificity.
  • Subgroup Analysis: Performance varied by HIV status, with lower sensitivity and specificity observed in HIV-positive individuals for both AI and radiologists.

Limitations and Future Work

The authors acknowledge several limitations:

  • Radiologist performance variability: The radiologists' performance did not consistently meet WHO targets, making it challenging to establish a single AI threshold that could satisfy both sensitivity and specificity requirements simultaneously.
  • Image artifacts: A shared CXR machine at two sites produced minor grid-like artifacts that, while ignorable by humans, were interpreted by the AI. This highlights the potential impact of real-world imaging imperfections on AI performance.
  • Site-level differences: Variations in TB positivity and abnormality rates across sites made it difficult to confidently attribute performance differences to specific factors.

Future work should focus on:

  • Population-specific tuning: Further investigation into customizing AI thresholds for specific populations and local resource constraints is recommended.
  • Real-world impact studies: More research is needed to confirm the impact of AI on reducing TB diagnosis delays and improving patient outcomes.
  • Broader artifact analysis: Investigating the impact of other real-world artifacts on AI performance is crucial for robust deployment.

Summary updated

Abstract

The CXR TB AI was noninferior to radiologists for active pulmonary TB triaging in a population with a high TB and HIV burden. Neither the TB AI nor the radiologists met WHO recommendations for sensitivity in the study population. AI can also be used to detect other CXR abnormalities in the same population.
BibTeX
@inproceedings{kazemzadeh2024prospective,
  author = {Sahar Kazemzadeh and Atilla P. Kiraly and Zaid Nabulsi and Nsala Sanjase and Minyoi M. Maimbolwa and Brian Shuma and Shahar Jamshy and Christina Chen and Arnav Agharwal and Charles T. Lau and Andrew Sellergren and D. I. Golden and Jin Yu and E H Wu and Yossi Matias and Katherine Chou and Greg S. Corrado and Shravya Shetty and Daniel Tse and Krishnan Eswaran and Yun Liu and Rory Pilgrim and Monde Muyoyeta and Shruthi Prabhakara},
  title = {{Prospective Multi-Site Validation of AI to Detect Tuberculosis and Chest X-Ray Abnormalities}},
  booktitle = {NEJM},
  year = {2024},
  doi = {10.1056/aioa2400018},
}
Figure