Abstract
Recent advancements have demonstrated the potential of using audio data—such as cough, breathing, and voice—for one-off COVID-19 screening. However, these methods typically fail to monitor disease progression or recovery over time. This study explores the use of longitudinal audio samples to predict COVID-19 progression using sequential deep learning. Analyzing crowdsourced data from 212 individuals over periods ranging from 5 to 385 days, we developed a tracking tool using Gated Recurrent Units (GRUs). Our model achieved an AUROC of 0.79 for detection and a high correlation (0.86) for tracking recovery trends, suggesting that longitudinal audio biomarkers are effective for remote health monitoring.
Introduction
Most AI-driven COVID-19 screening tools are designed for static, "one-off" detection. While useful for initial triage, these models do not account for the dynamic nature of the disease. Clinicians and patients require tools that can monitor whether a condition is worsening or if recovery is underway. A significant challenge in this domain is that audio characteristics vary widely between individuals; a "healthy" voice for one person might sound "congested" for another. By leveraging longitudinal data—repeated samples from the same individual—we can establish personal baselines and track the "audio dynamics" of an infection over time.
Methodology
The research team developed a deep learning pipeline specifically designed to handle sequential audio data through four key stages:
- Feature Extraction: Raw audio was converted into high-level mathematical embeddings using VGGish, a convolutional neural network (CNN) pretrained on large-scale audio datasets.
- Sequential Modeling: These embeddings were processed by Gated Recurrent Units (GRUs). Unlike standard networks, GRUs possess "memory," allowing them to capture temporal dependencies in how a person's voice or breath changes over a sequence of samples.
- Multitask Learning: To ensure the model focused on disease biomarkers rather than language-specific acoustic features, a multitask framework was implemented. The model was trained to simultaneously predict COVID-19 status and the speaker's language, forcing the hidden layers to ignore linguistic confounding factors.
- Trajectory Evaluation: The model's predicted "probability of infection" curve was compared against actual test result timelines using Dynamic Time Warping (DTW) and point-biserial correlation.
Data and Cohort
The study utilized data from the COVID-19 Sounds App (University of Cambridge), involving 212 longitudinal users (106 COVID-positive, 106 COVID-negative). Audio samples including coughs, breaths, and voice recordings were collected over 5 to 385 days. The cohort was linguistically diverse, featuring speakers of eight different languages, which necessitated the multitask learning approach to maintain model generalizability.
Key Insights and Results
A critical finding of the study was the visual and acoustic evidence of vocal cord changes during infection. In spectrograms, the researchers observed that during peak infection, the harmonics in a person's voice become "blurred," indicating a loss of vocal cord control. As the patient recovers, these harmonics become clearly separable again.
By modeling relative changes in audio rather than absolute values, the system effectively uses the individual's own healthy state as a control. Key performance metrics include:
- Superior Detection: The sequential model achieved an AUROC of 0.79, a sensitivity of 0.75, and a specificity of 0.71—a ~20% relative improvement over single-shot models.
- Recovery Tracking: The model was particularly adept at tracking the recovery phase, showing a 0.86 correlation with actual recovery reports in a subset of participants.
Limitations and Future Work
While promising, the study is limited by a relatively small longitudinal test cohort (21 individuals) and a reliance on self-reported test results. Additionally, other respiratory infections or emotional stress can cause similar acoustic changes. Future work will focus on larger, clinically validated datasets to better distinguish COVID-19 from other conditions and to develop participant-specific models for granular telemonitoring of various respiratory diseases.
Summary updated