Accelerating cough-based algorithms for pulmonary tuberculosis screening: Results from the CODA TB DREAM Challenge

Devan Jaganath, Solveig K. Sieberts, Mihaja Raberahona, Sophie Huddart, Larsson Omberg, Rivo Andry Rakotoarivelo, Issa N. Lyimo, Omar Lweno, Devasahayam Jesudas Christopher, Nguyen Viet Nhung, William Worodria, Charles Yu, Jhih-Yu Chen, Sz-Hau Chen, Tsai‐Min Chen, Chih-Han Huang, Kuei-Lin Huang, Filip Mulier, Daniel Rafter, Edward S.c. Shih, Yu Tsao, Hsuan-Kai Wang, Chih‐Hsun Wu, Christine Bachman, Stephen Burkot, Puneet Dewan, Sourabh Kulhare, Peter M. Small, Vijay Yadav, Simon Grandjean Lapierre, Grant Theron, Adithya CattamanchimedRxiv 2024 · DOI

TL;DR

The CODA TB DREAM Challenge demonstrated that open-access data competitions can rapidly accelerate the development of AI-based screening tools for global health. By combining cough acoustics with basic clinical data, researchers developed models that achieved over 80% sensitivity and 73% specificity for tuberculosis (TB) triage, nearing the World Health Organization’s targets for point-of-care screening.

What Problem Does This Paper Address?

Tuberculosis remains a leading cause of death worldwide, largely due to a massive "detection gap"—millions of people develop TB annually but are never diagnosed. While tools like chest X-rays and molecular testing exist, they require expensive infrastructure and specialized expertise often unavailable in primary care settings in low- and middle-income countries (LMICs).

Cough is a hallmark symptom of TB, and preliminary research suggests it has unique acoustic "signatures." However, the field has been held back by proprietary datasets and a lack of diverse data from different geographic regions. This paper describes a collaborative effort to build a specimen-free, low-cost triage tool using smartphone-recorded coughs.

Methods

The authors organized the CODA TB DREAM Challenge, a four-month open-science competition. Participants were tasked with two sub-challenges:

  1. Cough-only models: Classifying TB using only raw audio recordings.
  2. Combined models: Using audio plus basic clinical/demographic data (age, sex, BMI, smoking status, and symptoms like fever or night sweats).

To ensure transparency and reproducibility, teams submitted their models as Docker containers in the Open Neural Network Exchange (ONNX) format. The organizers then independently validated these models on a "held-out" test set that the participants never saw. The primary metrics were the Area Under the Receiver Operating Characteristic curve (AUROC) and a partial AUROC (pAUROC) focused on the high-sensitivity range (at least 80% sensitivity).

Datasets

The challenge utilized a newly established repository of data from 2,143 adults across seven high-burden countries: India, Madagascar, the Philippines, South Africa, Tanzania, Uganda, and Vietnam.

  • Audio: At least three solicited coughs per person, recorded via an Android app. The app used a CNN to isolate the 0.5-second peak sound of each cough.
  • Ground Truth: All participants underwent "gold standard" testing, including Xpert MTB/RIF Ultra and liquid/solid cultures.
  • Data Split: The dataset was split into training (50%), a "leaderboard" set for interim feedback (24%), and a final validation set (26%).

Key Insights and Ideas

  • The "Clinical Boost": While cough sounds alone showed promise, adding basic clinical data (like BMI and night sweats) significantly improved performance. This suggests that while the acoustic signal is real, it is most powerful when contextualized with the patient's physical state.
  • Bacterial Load Correlation: A fascinating "aha moment" was the discovery that the models' probability scores correlated with the bacterial load (measured by Xpert semi-quantitative levels). This implies the AI is picking up on physical changes in the lungs or airways caused by the severity of the infection.
  • Crowdsourcing Innovation: In just four months, the challenge produced multiple models that outperformed or matched previous studies that took years to complete. This highlights the power of "citizen science" and open-source frameworks in solving global health bottlenecks.
  • Demographic Sensitivity: Models performed better in males and HIV-negative individuals. This is likely because HIV-positive patients often have "paucibacillary" disease (lower bacterial loads), which may result in less distinct acoustic changes in their cough.

Key Results

  • Cough-Only: AUROCs ranged from 0.69 to 0.74. The top model achieved 55.5% specificity at 80% sensitivity.
  • Cough + Clinical: AUROCs improved to 0.78 to 0.83. The top-performing model reached 73.8% specificity at 80% sensitivity.
  • Target Benchmarks: Five of the six combined models reached the target pAUROC, approaching the WHO's Target Product Profile (TPP) for a TB triage test (which calls for ≥90% sensitivity and ≥70% specificity).
  • Algorithm Diversity: Winning teams used a variety of approaches, including Convolutional Neural Networks (CNNs), Gradient Boosting Decision Trees, and ensembles.

Limitations and Future Work

The authors acknowledge several hurdles for real-world deployment. First, the audio was limited to 0.5-second peak recordings; using the full duration of a cough might provide more diagnostic information. Second, the study focused on "solicited" coughs from symptomatic patients in clinics; it remains to be seen how these models perform in community-wide screening where many people may be asymptomatic.

Future work will focus on optimizing these models for different smartphone hardware and environmental noise levels, as well as exploring more complex "ensemble" architectures to push the accuracy closer to the WHO's 90% sensitivity threshold. To support this, the CODA TB dataset remains open-source for ongoing research.

Summary updated

Abstract

Abstract Importance Open-access data challenges have the potential to accelerate innovation in artificial-intelligence (AI)-based tools for global health. A specimen-free rapid triage method for TB is a global health priority. Objective To develop and validate cough sound-based AI algorithms for tuberculosis (TB) through the Cough Diagnostic Algorithm for Tuberculosis (CODA TB) DREAM challenge. Design In this diagnostic study, participating teams were provided cough-sound and clinical and demographic data. They were asked to develop AI models over a four-month period, and then submit the algorithms for independent validation. Setting Data was collected using smartphones from outpatient clinics in India, Madagascar, the Philippines, South Africa, Tanzania, Uganda, and Vietnam. Participants We included data from 2,143 adults who were consecutively enrolled with at least two weeks of cough. Data were randomly split evenly into training and test partitions. Exposures Standard TB evaluation was completed, including Xpert MTB/RIF Ultra and culture. At least three solicited coughs were recorded using the Hyfe Research app. Main Outcomes and Measures We invited teams to develop models using 1) cough sound features only and/or 2) cough sound features with routinely available clinical data to classify microbiologically confirmed TB disease. Models were ranked by area under the receiver operating characteristic curve (AUROC) and partial AUROC (pAUROC) to achieve at least 80% sensitivity and 60% specificity. Results Eleven cough models were submitted, as well as six cough-plus-clinical models. AUROCs for cough models ranged from 0.69-0.74, and the highest performing model achieved 55.5% specificity (95% CI 47.7-64.2) at 80% sensitivity. The addition of clinical data improved AUROCs (range 0.78-0.83), five of the six submitted models reached the target pAUROC, and highest performing model had 73.8% (95% CI 60.8-80.0) specificity at 80% sensitivity. In post-challenge subgroup analyses, AUROCs varied by country, and was higher among males and HIV-negative individuals. The probability of TB classification correlated with Xpert Ultra semi-quantitative levels. Conclusions and Relevance In a short period, new and independently validated cough-based TB algorithms were developed through an open-source and transparent process. Open-access data challenges can rapidly advance and improve AI-based tools for global health. Key Points Question Can an open-access data challenge support the rapid development of cough-based artificial intelligence (AI) algorithms to screen for tuberculosis (TB)? Findings In this diagnostic study, teams were provided well-characterized cough sound data from seven countries, and developed and submitted AI models for independent validation. Multiple models that combined clinical and cough data achieved the target accuracy of at least 80% sensitivity and 60% specificity to classify microbiologically-confirmed TB. Meaning Cough-based AI models have promise to support point-of-care TB screening, and open-access data challenges can accelerate the development of AI-based tools for global health.
BibTeX
@misc{jaganath2024accelerating,
  author = {Devan Jaganath and Solveig K. Sieberts and Mihaja Raberahona and Sophie Huddart and Larsson Omberg and Rivo Andry Rakotoarivelo and Issa N. Lyimo and Omar Lweno and Devasahayam Jesudas Christopher and Nguyen Viet Nhung and William Worodria and Charles Yu and Jhih-Yu Chen and Sz-Hau Chen and Tsai‐Min Chen and Chih-Han Huang and Kuei-Lin Huang and Filip Mulier and Daniel Rafter and Edward S.C. Shih and Yu Tsao and Hsuan-Kai Wang and Chih‐Hsun Wu and Christine Bachman and Stephen Burkot and Puneet Dewan and Sourabh Kulhare and Peter M. Small and Vijay Yadav and Simon Grandjean Lapierre and Grant Theron and Adithya Cattamanchi},
  title = {{Accelerating cough-based algorithms for pulmonary tuberculosis screening: Results from the CODA TB DREAM Challenge}},
  year = {2024},
  doi = {10.1101/2024.05.13.24306584},
  howpublished = {medRxiv},
}
Figure