Interpretable Acoustic Representation Learning on Breathing and Speech Signals for COVID-19 Detection

Debottam Dutta, Debarpan Bhattacharya, Sriram Ganapathy, Amir H. Poorjam, Deepak Mittal, Maneesh Singh

TL;DR

This paper introduces an interpretable representation learning framework for detecting COVID-19 from raw audio recordings of speech and breathing. By combining a learnable filterbank of cosine-modulated Gaussian functions with a self-attention relevance weighting mechanism, the model outperforms standard audio baselines while revealing exactly which frequency bands matter most. Crucially, the study highlights that breathing sounds serve as a highly universal physiological biomarker for transfer learning, whereas speech signals are heavily constrained by regional and linguistic differences.

What Problem Does This Paper Address?

Using audio signals (like coughs, speech, and breathing) for automated medical diagnosis holds immense promise for non-invasive screening. However, biomedical applications demand a high degree of interpretability—clinicians need to know why a model makes a decision.

Traditional approaches rely on hand-crafted acoustic features (such as jitter or shimmer), which require tedious domain-specific engineering and extensive experimentation. Conversely, modern deep learning models can learn representations directly from raw data, but they function as uninterpretable "black boxes" and struggle with the high temporal variability of audio. This paper bridges that gap by proposing an audio front-end that learns optimal time-frequency representations directly from raw waveforms while remaining physically interpretable as a bank of smooth band-pass filters.

Methods

The authors propose a novel neural network front-end consisting of two primary components:

  1. Acoustic Filterbank Layer: Instead of using standard fixed Mel-filterbanks, the raw audio is passed through a 1-D convolutional layer where the filters are parameterized as cosine-modulated Gaussian functions. The Gaussian component acts as a low-pass filter, and the cosine modulation shifts it to a specific band-pass frequency. The only learnable parameters are the center frequencies ($\mu_i$), ensuring that the network learns a physically meaningful, smooth filterbank layout (constant-Q filters) directly from data.
  2. Relevance Weighting Module: The output of the filterbank is squared, pooled, and log-compressed to form a spectrogram. This spectrogram is then fed into a self-attention-style feed-forward network that computes a "relevance weight" for each time-frequency bin. This dynamically highlights the acoustic regions most critical for the downstream task.

The entire pipeline is illustrated below:

Figure 1: Overall pipeline for filter learning. The input signal is passed through a 1-D convolutional layer where we use Gaus- sian set of filters. The relevance weighting module consists of a fully connected layer followed by a sigmoid function. Two maps are also shown as output of convolution operation and the cor- responding learned mask. Blocks in green depict a learnable function whereas the blue blocks depict fixed functions.

For the downstream classification, the relevance-weighted representations are augmented with delta and double-delta features and passed to a 2-layer Bidirectional LSTM (BLSTM) network to predict COVID-19 status.

Additionally, the authors explore two transfer learning paradigms using a larger dataset: Supervised Pre-training and Self-Supervised Pre-training via Contrastive Predictive Coding (CPC).

Datasets

The models were developed and evaluated using two distinct crowd-sourced datasets:

  • COVID-19 Sounds Dataset: A large-scale dataset crowd-sourced primarily in the UK, containing 53,449 audio samples from over 36,000 participants. The authors selected a subset of 44.1 kHz .wav files (split into ~1,800 training and ~450 validation samples per modality) to use exclusively for pre-training.
  • Coswara Dataset: A dataset crowd-sourced by the Indian Institute of Science, containing recordings from over 1,500 participants. A subset of 965 participants, formatted into a 5-fold cross-validation setup from the Second DiCOVA challenge, was used as the target dataset for evaluation.

Key Insights and Ideas

  • The Universality of Breathing vs. Speech: The most striking finding is how differently transfer learning behaves across audio modalities. Supervised pre-training on the UK-centric COVID-19 Sounds dataset followed by fine-tuning on the Indian Coswara dataset yielded a massive performance boost for breathing sounds, but actually degraded performance for speech sounds. The authors hypothesize that speech is deeply tied to culture, accent, and language, making cross-continental transfer difficult. Breathing, being a purely physiological function, transfers universally.
  • Data-Driven Filter Distribution: When analyzing the learned center frequencies, the model automatically allocated more filters to the low-frequency regions (0-4 kHz) for speech. For breathing, the model deviated significantly from the standard Mel-scale, stretching out filters in the mid-to-high frequency ranges to capture subtle respiratory anomalies.
Learned center frequencies for breathing and speech compared to the standard Mel scale.
  • Visualizing Relevance: The relevance weighting mechanism acts as an interpretable mask. As seen in the spectrogram visualizations, it actively enhances high-frequency regions in breathing samples, confirming that upper-harmonic acoustic energy contains vital clues for COVID-19 detection.
Figure 2: (a) Mel spectrogram, (b) the learned spectrogram and (c) relevance weighted spectrogram of a breathing sample.

Key Results

Performance was evaluated using the Area Under the ROC Curve (AUC %). The proposed CosGauss-relev model trained directly on the target data outperformed standard baselines like Mel-Spectrograms, SincNet, and LEAF across both modalities.

  • Breathing Modality: The baseline Mel-spectrogram achieved an average AUC of 77.4%. The proposed model with direct supervised learning achieved 78.4%. When leveraging supervised pre-training and fine-tuning (CosGauss-relev-pretr-fine), the performance jumped significantly to 80.5% AUC.
  • Speech Modality: Direct supervised learning with the proposed model achieved the highest overall performance at 81.5% AUC, outperforming the Mel-spectrogram baseline (79.6%) and other learnable front-ends like LEAF (60.6%).

The complete breakdown across all 5 evaluation folds can be seen in Table 1:

Table 1: Comparison of the performance of the COVID-19 detection models using different signal representations on breathing and speech modalities in terms of the AUC(%).

Limitations and Future Work

While the framework provides excellent interpretability and performance, a clear limitation is its sensitivity to domain shifts in speech signals caused by geographic and linguistic diversity. Pre-training on speech from one region does not inherently translate to another without localized data.

Future work could explore domain adaptation techniques to mitigate these linguistic barriers or expand the self-supervised learning framework to encompass broader, multi-modal respiratory acoustic properties.

Summary updated

Abstract

In this paper, we describe an approach for representation learning of audio signals for the task of COVID-19 detection. The raw audio samples are processed with a bank of 1-D convolutional filters that are parameterized as cosine modulated Gaussian functions. The choice of these kernels allows the interpretation of the filterbanks as smooth band-pass filters. The filtered outputs are pooled, log-compressed and used in a self-attention based relevance weighting mechanism. The relevance weighting emphasizes the key regions of the time-frequency decomposition that are important for the downstream task. The subsequent layers of the model consist of a recurrent architecture and the models are trained for a COVID-19 detection task. In our experiments on the Coswara data set, we show that the proposed model achieves significant performance improvements over the baseline system as well as other representation learning approaches. Further, the approach proposed is shown to be uniformly applicable for speech and breathing signals and for transfer learning from a larger data set.
BibTeX
@misc{dutta2022interpretable,
  author = {Debottam Dutta and Debarpan Bhattacharya and Sriram Ganapathy and Amir H. Poorjam and Deepak Mittal and Maneesh Singh},
  title = {{Interpretable Acoustic Representation Learning on Breathing and Speech Signals for COVID-19 Detection}},
  year = {2022},
  eprint = {2206.13365},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2206.13365},
}
Figure