CAD4TB software updates: different triaging thresholds require caution by users and regulation by authorities

Jana Fehr, Resign Gunda, Mark J. Siedner, Willem A. Hanekom, T. Ndung ́u, Alison D. Grant, Christoph Lippert, Emily WongThe International Journal of Tuberculosis and Lung Disease 2023 · DOI

TL;DR

New versions of CAD4TB, a widely used AI tool for tuberculosis screening, show similar overall performance but significantly different score distributions. This necessitates careful re-evaluation of triaging thresholds with each update to avoid missing cases or over-testing, highlighting a need for better manufacturer transparency and regulatory oversight.

What Problem Does This Paper Address?

The World Health Organization (WHO) has recommended systematic tuberculosis (TB) screening using digital chest X-rays (CXR) interpreted by computer-aided diagnostic (CAD) algorithms. CAD4TB is a popular tool that assigns a score (0-100) indicating the likelihood of TB. Screening programs must set a "triaging threshold" above which individuals are sent for sputum testing. However, these thresholds are not universal and need adjustment based on local factors. The challenge is that CAD4TB is updated annually, and there's no clear guidance on how to adapt triaging thresholds for new versions, potentially leading to errors in screening programs. This paper investigates whether new versions of CAD4TB maintain consistent performance and how their score distributions change, impacting the choice of triaging thresholds.

Methods

The researchers evaluated the performance of three versions of CAD4TB (v5, v6, and v7) using retrospective CXR data from a community-based TB screening study in rural South Africa. They compared the overall performance (measured by Area Under the Curve - AUC) of the CAD4TB versions against a senior radiologist's interpretation. Crucially, they also analyzed the distribution of CAD4TB scores for all CXRs and examined how different triaging thresholds affected the number of participants referred for sputum testing and the number of missed TB cases. They identified specific thresholds for each CAD4TB version that matched the radiologist's sensitivity for TB detection and compared the resulting specificity and the proportion of participants tested.

Datasets

The study utilized retrospective CXR data from 9,912 participants in a community-based multi-morbidity study in rural KwaZulu-Natal, South Africa. Sputum test results (Xpert® MTB/RIF Ultra and MGITM liquid culture) were available for 4,976 participants. CXRs were interpreted by a senior radiologist, and these interpretations served as the reference standard.

Key Insights and Ideas

  • Software updates aren't always "better" in a simple way: While newer versions of CAD4TB showed similar overall diagnostic accuracy (AUC), the way they assign scores changed dramatically. This means a threshold that worked well in an older version might be completely inappropriate for a newer one.
  • Score distribution is critical: The median CAD4TB scores varied significantly between versions (e.g., v5 median 28, v7 median 11). This shift means that a fixed numerical threshold (like 40) can lead to vastly different numbers of people being tested and different rates of missed TB cases across versions.
  • No "one-size-fits-all" threshold: The study found that no single threshold for any CAD4TB version met the WHO's target product profile for high sensitivity and specificity. This reinforces the need for context-specific threshold selection.
  • Lack of transparency is a major issue: Manufacturers don't clearly communicate the changes in their algorithms or the data used for training new versions. This makes it incredibly difficult for screening programs to know if and how they should adjust their triaging thresholds.

Key Results

  • The overall performance (AUC) of CAD4TB v5, v6, and v7 was similar, ranging from 0.78 to 0.80.
  • However, the distribution of CAD4TB scores varied significantly across versions, with median scores of 28 (v5), 35 (v6), and 11 (v7).
  • Using a triaging threshold of 40, sensitivities ranged from 66.7% (v7) to 88.9% (v6), and specificities from 33.3% (v6) to 84.6% (v7).
  • When matching the radiologist's sensitivity (around 80.8%), the thresholds that achieved this were v5: 40, v6: 47, and v7:
  • At these matched sensitivities, v7 had lower specificity than v5 and v6.
  • These different thresholds led to substantial variations in the percentage of participants requiring sputum testing (ranging from 23.7% to 33.5% for the matched thresholds) and the percentage of missed microbiologically confirmed TB cases (ranging from 11.1% to 33.3%).

Limitations and Future Work

The authors acknowledge that the lack of transparency from CAD developers regarding algorithmic changes makes it difficult for end-users to adapt. They call for regulation requiring companies to communicate these changes and provide guidance for effective adoption. They also suggest that ongoing vigilance and performance auditing of CAD software versions should be a requirement for WHO authorization. The study's findings on score distribution shifts are crucial for other AI interpretation algorithms used in TB screening, and further research is needed to determine if these variations are applicable across different platforms. The authors emphasize the need for pilot studies to find optimal triaging thresholds for each new software update, or at least a clear method for retrospective analysis to guide re-adjustment.

Summary updated

Abstract

different triaging thresholds require caution by users and regulation by authorities
BibTeX
@article{fehr2023cad,
  author = {Jana Fehr and Resign Gunda and Mark J. Siedner and Willem A. Hanekom and T. Ndung ́u and Alison D. Grant and Christoph Lippert and Emily Wong},
  title = {{CAD4TB software updates: different triaging thresholds require caution by users and regulation by authorities}},
  journal = {The International Journal of Tuberculosis and Lung Disease},
  year = {2023},
  doi = {10.5588/ijtld.22.0437},
}
Figure