TL;DR
This paper introduces an interpretable representation learning framework for detecting COVID-19 from raw audio recordings of speech and breathing. By combining a learnable filterbank of cosine-modulated Gaussian functions with a self-attention relevance weighting mechanism, the model outperforms standard audio baselines while revealing exactly which frequency bands matter most. Crucially, the study highlights that breathing sounds serve as a highly universal physiological biomarker for transfer learning, whereas speech signals are heavily constrained by regional and linguistic differences.
What Problem Does This Paper Address?
Using audio signals (like coughs, speech, and breathing) for automated medical diagnosis holds immense promise for non-invasive screening. However, biomedical applications demand a high degree of interpretability—clinicians need to know why a model makes a decision.
Traditional approaches rely on hand-crafted acoustic features (such as jitter or shimmer), which require tedious domain-specific engineering and extensive experimentation. Conversely, modern deep learning models can learn representations directly from raw data, but they function as uninterpretable "black boxes" and struggle with the high temporal variability of audio. This paper bridges that gap by proposing an audio front-end that learns optimal time-frequency representations directly from raw waveforms while remaining physically interpretable as a bank of smooth band-pass filters.
Methods
The authors propose a novel neural network front-end consisting of two primary components:
- Acoustic Filterbank Layer: Instead of using standard fixed Mel-filterbanks, the raw audio is passed through a 1-D convolutional layer where the filters are parameterized as cosine-modulated Gaussian functions. The Gaussian component acts as a low-pass filter, and the cosine modulation shifts it to a specific band-pass frequency. The only learnable parameters are the center frequencies ($\mu_i$), ensuring that the network learns a physically meaningful, smooth filterbank layout (constant-Q filters) directly from data.
- Relevance Weighting Module: The output of the filterbank is squared, pooled, and log-compressed to form a spectrogram. This spectrogram is then fed into a self-attention-style feed-forward network that computes a "relevance weight" for each time-frequency bin. This dynamically highlights the acoustic regions most critical for the downstream task.
The entire pipeline is illustrated below:
For the downstream classification, the relevance-weighted representations are augmented with delta and double-delta features and passed to a 2-layer Bidirectional LSTM (BLSTM) network to predict COVID-19 status.
Additionally, the authors explore two transfer learning paradigms using a larger dataset: Supervised Pre-training and Self-Supervised Pre-training via Contrastive Predictive Coding (CPC).
Datasets
The models were developed and evaluated using two distinct crowd-sourced datasets:
- COVID-19 Sounds Dataset: A large-scale dataset crowd-sourced primarily in the UK, containing 53,449 audio samples from over 36,000 participants. The authors selected a subset of 44.1 kHz
.wavfiles (split into ~1,800 training and ~450 validation samples per modality) to use exclusively for pre-training. - Coswara Dataset: A dataset crowd-sourced by the Indian Institute of Science, containing recordings from over 1,500 participants. A subset of 965 participants, formatted into a 5-fold cross-validation setup from the Second DiCOVA challenge, was used as the target dataset for evaluation.
Key Insights and Ideas
- The Universality of Breathing vs. Speech: The most striking finding is how differently transfer learning behaves across audio modalities. Supervised pre-training on the UK-centric COVID-19 Sounds dataset followed by fine-tuning on the Indian Coswara dataset yielded a massive performance boost for breathing sounds, but actually degraded performance for speech sounds. The authors hypothesize that speech is deeply tied to culture, accent, and language, making cross-continental transfer difficult. Breathing, being a purely physiological function, transfers universally.
- Data-Driven Filter Distribution: When analyzing the learned center frequencies, the model automatically allocated more filters to the low-frequency regions (0-4 kHz) for speech. For breathing, the model deviated significantly from the standard Mel-scale, stretching out filters in the mid-to-high frequency ranges to capture subtle respiratory anomalies.
- Visualizing Relevance: The relevance weighting mechanism acts as an interpretable mask. As seen in the spectrogram visualizations, it actively enhances high-frequency regions in breathing samples, confirming that upper-harmonic acoustic energy contains vital clues for COVID-19 detection.
Key Results
Performance was evaluated using the Area Under the ROC Curve (AUC %). The proposed CosGauss-relev model trained directly on the target data outperformed standard baselines like Mel-Spectrograms, SincNet, and LEAF across both modalities.
- Breathing Modality: The baseline Mel-spectrogram achieved an average AUC of 77.4%. The proposed model with direct supervised learning achieved 78.4%. When leveraging supervised pre-training and fine-tuning (
CosGauss-relev-pretr-fine), the performance jumped significantly to 80.5% AUC. - Speech Modality: Direct supervised learning with the proposed model achieved the highest overall performance at 81.5% AUC, outperforming the Mel-spectrogram baseline (79.6%) and other learnable front-ends like LEAF (60.6%).
The complete breakdown across all 5 evaluation folds can be seen in Table 1:
Limitations and Future Work
While the framework provides excellent interpretability and performance, a clear limitation is its sensitivity to domain shifts in speech signals caused by geographic and linguistic diversity. Pre-training on speech from one region does not inherently translate to another without localized data.
Future work could explore domain adaptation techniques to mitigate these linguistic barriers or expand the self-supervised learning framework to encompass broader, multi-modal respiratory acoustic properties.
Summary updated