When we breathe, cough, or speak, our respiratory tract acts as a dynamic acoustic instrument. Pathological changes—such as the airway inflammation of asthma, the fluid accumulation of pneumonia, or the tissue changes from viral infections like COVID-19—alter the physical properties of this instrument. These changes modify airflow dynamics and tissue vibrations, leaving distinct, often sub-audible signatures in the resulting sound waves.
A rapidly growing body of research leverages machine learning to transform these non-invasive acoustic signals into objective diagnostic tools. By curating specialized datasets of respiratory and environmental sounds, researchers are moving away from subjective clinical listening toward automated, high-dimensional pattern recognition.
From Waveforms to High-Dimensional Features
The core challenge of acoustic diagnostics lies in translating a continuous, non-stationary audio signal into a representation that a model can classify. Early approaches relied heavily on hand-crafted feature engineering to capture different dimensions of sound. For instance, Stavros Ntalampiras et al. demonstrated in A multidomain approach for automatic home environmental sound classification that combining Mel-Frequency Cepstral Coefficients (MFCCs) with spectral features and wavelet packets yields highly accurate environmental sound classification by capturing both coarse spectral envelopes and fine temporal transients.
As deep learning matured, the field shifted toward models that learn representations directly from data, either via time-frequency representations (spectrograms) or raw audio waveforms.
Raw Audio Waveform ──► Spectrogram (2D Time-Frequency) ──► Audio Spectrogram Transformer (AST) / CNN ──► Diagnostic Label
This evolution is visible across several diagnostic domains:
- Cough Analysis: Hwan Ing Hee et al. applied machine learning to voluntary cough sounds in Development of Machine Learning for Asthmatic and Healthy Voluntary Cough Sounds: A Proof of Concept Study to distinguish asthmatic from healthy children. To capture the complex temporal progression of a cough, B T Balamurali et al. utilized Bidirectional Long Short-Term Memory (BiLSTM) networks in Deep Neural Network-Based Respiratory Pathology Classification Using Cough Sounds, allowing the model to process acoustic context both forward and backward in time to isolate lower and upper respiratory tract infections.
- Stethoscope and Breath Sounds: Instead of relying on manual feature selection, Paridhi Mundra et al. introduced VoxMed in VoxMed: one-step respiratory disease classifier using digital stethoscope sounds, combining an Audio Spectrogram Transformer (AST) with a 1D CNN to capture global context from digital stethoscope recordings. Similarly, Jiayuan She et al. developed a lightweight deep learning architecture in Detection of disease on nasal breath sound by new lightweight architecture: Using COVID-19 as an example to classify COVID-19 from simple nasal breathing sounds recorded on smartphones.
- Transient Bioacoustics: The utility of raw waveform processing extends beyond human health. In bioacoustics, Maxence Ferrari et al. showed in End to end raw audio deep learning of transients, application to bioacoustics. that end-to-end deep learning on raw audio can successfully classify highly transient signals like dolphin clicks without requiring manual spectrogram tuning.
The underlying reason these deep architectures work is their ability to mirror the hierarchical processing of complex sounds. This parallel is supported by neuroimaging work from Tzu-Han Zoe Cheng et al. in Similar Hierarchical Representation of Speech and Other Complex Sounds In the Brain and Deep Residual Networks: An MEG Study, which found that deep residual networks organize and specialize acoustic representations in a manner strikingly similar to the human auditory cortex.
The Role of Curation and Real-World Noise
An algorithm is only as robust as the acoustic environment it is trained on. Early diagnostic models often suffered from overfitting because they were trained on clean, studio-quality recordings. Real-world deployment requires models to perform in noisy home or hospital environments.
To address this, the community has focused heavily on curating standardized, diverse benchmark datasets. For example, Neeraj Kumar Sharma 0001 et al. organized the Second DiCOVA Challenge, detailed in The Second Dicova Challenge: Dataset and Performance Analysis for Diagnosis of Covid-19 Using Acoustics, providing a large-scale, crowdsourced acoustic dataset of coughs, breathing, and speech to establish rigorous baseline benchmarks for COVID-19 diagnosis.
However, moving from controlled clinical environments to home care introduces significant challenges. Heidi Cleverley-Leblanc et al. explored this boundary in Enhancing Pediatric Asthma Homecare Management: The Potential of Deep Learning Associated with Spirometry-Labelled Data. By pairing lung sound recordings with objective spirometry labels, the authors observed a performance drop during out-of-hospital validation, highlighting how device variability and ambient noise can obscure subtle pathological acoustic features.
This need for rigorous negative sampling and environmental baselines spans other acoustic classification domains as well:
- In structural/environmental monitoring, Marcin Detyniecki developed Crashzam in Crashzam: Sound-based Car Crash Detection, emphasizing the necessity of training models on "negative mimics"—common in-car sounds that resemble crashes—to prevent false positives.
- In wildlife monitoring, Joeri A. Zwerts et al. introduced a massive semi-natural sanctuary dataset in Introducing a Central African Primate Vocalisation Dataset for Automated Species Classification to train models capable of isolating primate calls from dense jungle background noise.
- For security and attribution, Anton Firc et al. created the STOPA dataset in STOPA: A Dataset of Systematic VariaTion Of DeePfake Audio for Open-Set Source Tracing and Attribution, isolating how subtle acoustic artifacts from different vocoders can be traced back to specific generative models.
Contextual and Multimodal Fusion
Why limit a model to a single audio type? A patient suffering from a respiratory illness does not just have an altered cough; their breathing cadence changes, and their vocal cords may exhibit swelling that alters their speech.
Recent breakthroughs demonstrate that fusing multiple acoustic modalities drastically improves diagnostic accuracy. Adria Mallol-Ragolta et al. investigated this in Multi-Type Outer Product-Based Fusion of Respiratory Sounds for Detecting COVID-19, utilizing an outer-product mathematical fusion to combine cough, breath, and speech recordings. Their findings revealed that combining breath and speech sounds yielded a higher area under the curve (AUC) than analyzing any single sound type in isolation, as the model could cross-reference spectral changes across different physiological mechanisms.
┌───────────────┐
│ Cough Audio │ ──┐
└───────────────┘ │
┌───────────────┐ ├─► Outer-Product Fusion ─► Multimodal Classifier ─► Diagnosis
│ Breath Audio │ ──┤
└───────────────┘ │
┌───────────────┐ │
│ Speech Audio │ ──┘
└───────────────┘
Beyond combining different sounds, incorporating non-acoustic contextual metadata helps ground the model's predictions. In Cough-Based COVID-19 Detection with Contextual Attention Convolutional Neural Networks and Gender Information, Adria Mallol-Ragolta et al. demonstrated that integrating gender information into a ResNet18 architecture significantly boosts performance. Because baseline vocal tract lengths and lung capacities vary systematically by gender, providing this context allows the network's attention mechanism to calibrate its acoustic expectations.
Finally, the field is moving toward interactive, open-ended diagnostic reasoning. Rather than outputting a simple binary classification, modern architectures aim to explain why a diagnosis is given. Tsai-Ning Wang et al. advanced this frontier by introducing CaReAQA and the CaReSound dataset in CaReAQA: A Cardiac and Respiratory Audio Question Answering Model for Open-Ended Diagnostic Reasoning. This audio-language model ingests clinical cardiac and respiratory sounds alongside textual questions, enabling open-ended diagnostic reasoning that bridges the gap between raw acoustic waveforms and human-interpretable clinical insights.
Key Foundational Work
A multidomain approach for automatic home environmental sound classification (2010) — contributed 1 concept elements to this direction.
A Dataset and Taxonomy for Urban Sound Research (2014) — contributed 1 concept elements to this direction.
On the acoustic environment of a neonatal intensive care unit: initial description, and detection of equipment alarms (2014) — contributed 1 concept elements to this direction.