Exploring Longitudinal Cough, Breath, and Voice Data for COVID-19 Progression Prediction via Sequential Deep Learning: Model Development and Validation

Ting Dang, Jing Han, Xia Tong, Dimitris Spathis, Erika Bondareva, Chloë Brown, Jagmohan Chauhan, Andreas Grammenos, Apinan Hasthanasombat, R. Andrés Floto, Pietro Cicuta, Cecilia Mascolo

Abstract

An audio-based COVID-19 progression monitoring system was developed using deep learning techniques, with strong performance showing high consistency between the predicted trajectory and the test results over time, especially for recovery trend predictions. This has good potential in the postpeak and postpandemic era that can help guide medical treatment and optimize hospital resource allocations. The changes in longitudinal audio samples, referred to as audio dynamics, are associated with COVID-19 progression; thus, modeling the audio dynamics can potentially capture the underlying disease progression process and further aid COVID-19 progression prediction. This framework provides a flexible, affordable, and timely tool for COVID-19 tracking, and more importantly, it also provides a proof of concept of how telemonitoring could be applicable to respiratory diseases monitoring, in general.

Introduction

While many AI models have been developed to detect COVID-19 from a single audio sample, they typically function as "one-and-done" screening tools, failing to capture the temporal evolution of the disease. In clinical or home-quarantine settings, knowing if someone has the virus is only the first step; understanding whether they are worsening or recovering is critical for resource management and timely medical intervention.

Researchers identify a "paucity of work" on continuous monitoring. They hypothesize that COVID-19 leaves a distinct signature on the respiratory system that changes over the course of the infection—specifically affecting vocal cord vibration and lung function—which can be captured through longitudinal audio biomarkers.

Methods

The researchers employed a sequential deep learning approach to capture the "audio dynamics" of the infection using a multi-stage pipeline:

  1. Feature Extraction: The team utilized VGGish, a pre-trained convolutional neural network (CNN) from Google, to convert audio spectrograms into high-level feature embeddings.
  2. Sequential Modeling: These embeddings were processed by Gated Recurrent Units (GRUs). GRUs were selected over standard LSTMs for their efficiency and reduced risk of overfitting in limited data size regimes.
  3. Multi-task Learning: To prevent "language bias" (e.g., the model associating specific languages with higher infection rates), the framework was trained to simultaneously detect COVID-19 and recognize the speaker's language, forcing the network to isolate disease biomarkers from linguistic features.
  4. Data Augmentation: Techniques such as time-inverse augmentation were used to simulate diverse disease progression patterns and bolster the training set.

Datasets

The study utilized a crowdsourced dataset collected via a mobile app developed by the University of Cambridge. From an initial pool of 3,845 healthy and 1,456 COVID-positive participants, the researchers identified a balanced cohort of 212 longitudinal users (106 positive, 106 negative) who provided at least five samples over periods ranging from 5 to 385 days. Each participant recorded three voluntary coughs, 3–5 deep breaths, and read a short sentence on the screen three times.

Key Insights: The Separability of Harmonics

A critical finding in this research is the visual evidence of disease progression in audio spectrograms. During peak infection, the fundamental frequency and harmonics of a participant's voice often become blurred and non-separable, indicating a loss of control over vocal cord vibration. As the patient recovers, these harmonics become distinct again.

Spectrogram of a COVID-positive audio sample (2020-11-18) showing blurred harmonics.
Spectrogram of the same participant's audio sample after recovery (2020-11-22) showing distinct harmonics.

By using a sequence of samples, the model establishes a personalized baseline for each individual. This allows it to detect relative changes that are more indicative of disease progression than absolute values, which vary widely between healthy individuals.

Key Results

  • Detection Performance: The sequential model achieved an AUROC of 0.79, significantly outperforming "single-shot" models (AUROC 0.66) and baseline models that simply averaged past features (AUROC 0.60).
  • Progression Tracking: The predicted probability of infection showed a 0.75 correlation with actual longitudinal test results across the entire test cohort.
  • Recovery Prediction: In a subset of 12 participants who specifically reported recovery, the correlation between the model’s predicted trajectory and their clinical course reached 0.86.
  • Temporal Alignment: Using Dynamic Time Warping (DTW), the researchers confirmed that the model's audio-based timeline closely matched the biological reality of the disease, often predicting recovery trends before self-reported negative test results.

Limitations and Future Work

Despite promising results, the authors note several hurdles. The sample size of high-quality longitudinal data remains relatively small (212 individuals), and the "ground truth" labels relied on self-reported test results, which may contain inaccuracies. Furthermore, it is not yet clear if the model can distinguish COVID-19 progression from other respiratory illnesses like influenza or asthma.

Future research will focus on developing participant-specific models that can be fine-tuned to an individual's unique voice and breathing patterns, further increasing the precision of remote telemonitoring for a variety of respiratory conditions.

Summary updated

Abstract

An audio-based COVID-19 progression monitoring system was developed using deep learning techniques, with strong performance showing high consistency between the predicted trajectory and the test results over time, especially for recovery trend predictions. This has good potential in the postpeak and postpandemic era that can help guide medical treatment and optimize hospital resource allocations. The changes in longitudinal audio samples, referred to as audio dynamics, are associated with COVID-19 progression; thus, modeling the audio dynamics can potentially capture the underlying disease progression process and further aid COVID-19 progression prediction. This framework provides a flexible, affordable, and timely tool for COVID-19 tracking, and more importantly, it also provides a proof of concept of how telemonitoring could be applicable to respiratory diseases monitoring, in general.
BibTeX
@article{dang2022exploring,
  author = {Ting Dang and Jing Han and Xia Tong and Dimitris Spathis and Erika Bondareva and Chloë Brown and Jagmohan Chauhan and Andreas Grammenos and Apinan Hasthanasombat and R. Andrés Floto and Pietro Cicuta and Cecilia Mascolo},
  title = {{Exploring Longitudinal Cough, Breath, and Voice Data for COVID-19 Progression Prediction via Sequential Deep Learning: Model Development and Validation}},
  journal = {Journal of Medical Internet Research},
  year = {2022},
  doi = {10.2196/37004},
}
Figure