Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Data

Hamza Mahdi, Eptehal Nashnoush, Rami Saab, Arjun Balachandar, Rishit Dagli, Lucas X. Perri, Houman KhosravaniarXiv 2024 · DOI · arXiv

Abstract

This study evaluates the performance of various deep learning architectures—including CNNs and Transformers—on audio classification tasks within data-constrained clinical environments. Using a dataset of stroke-recovery audio recordings, we compare different preprocessing techniques and transfer learning strategies. Our findings indicate that optimized CNNs, such as DenseNet and ConvNeXt, can match or outperform state-of-the-art Transformers when data is scarce, particularly when leveraging specific pre-training priors and contrastive loss functions.

Introduction

In artificial intelligence, the prevailing paradigm is that "more data" leads to better models. However, in clinical settings, high-quality data is a scarce resource. Collecting audio from patients, such as stroke survivors with speech or swallowing difficulties (dysphagia), is resource-intensive and constrained by ethical and privacy requirements. Most state-of-the-art audio classifiers are trained on massive datasets like YouTube’s AudioSet, raising the question: do these "big data" models translate effectively to "small data" clinical environments? This research bridges this gap by identifying the optimal combinations of architecture and preprocessing for non-invasive, low-cost clinical assessment using limited patient data.

Methodology

The researchers conducted a comparative analysis of several architectures, including CNNs (DenseNet, ConvNeXt), Transformers (ViT, Swin, AST), and pre-trained audio models (YAMNet, VGGish, Trill).

Overview of the clinical audio classification pipeline, from data collection to model prediction.

Technical Innovations

The study introduced several key methodological variations to optimize performance:

  1. Hybrid Loss Functions: For DenseNet, a hybrid loss combining standard Cross-Entropy with Contrastive Loss was utilized to improve representation learning in the face of class imbalances.
  2. Preprocessing Representations: Three audio representations were compared: standard monochrome Mel-spectrograms, three-channel RGB Mel-spectrograms (to leverage ImageNet pre-training), and Superlet transforms for enhanced time-frequency resolution.
  3. Multistage Transfer Learning: Models underwent a training pipeline starting with general image data (ImageNet), followed by general audio (US8K or ESC50), and concluding with fine-tuning on clinical data.

Datasets

The evaluation utilized two novel clinical datasets alongside public benchmarks:

  • Dataset NIHSS: Audio of stroke patients performing speech tasks based on the National Institutes of Health Stroke Scale.
  • Dataset Vowel: Sustained vowel sounds (/a/, /e/, /i/, /o/, /u/) used for detecting swallowing disorders.
  • Public Benchmarks: ImageNet, AudioSet, US8K (urban sounds), and ESC50 (environmental sounds).

Results and Discussion

The study revealed that the choice of preprocessing is heavily dependent on the model's pre-training "priors." Models pre-trained on ImageNet performed significantly better with RGB spectrograms, while those pre-trained on AudioSet preferred monochrome inputs.

Performance metrics across various model architectures and preprocessing methods.

Key performance highlights include:

  • CNN Resilience: ConvNeXt (RGB) achieved a high AUC of 0.91 and a balanced F1 score of 0.84, demonstrating that CNNs' inductive biases are highly effective in low-data regimes.
  • Contrastive Learning: The DenseNet-Contrastive model (pre-trained on US8K) achieved a perfect Specificity (1.0), effectively eliminating false positives for patients without the condition.
  • Transformer Efficiency: The Audio Spectrogram Transformer (AST) reached peak performance in only 6 epochs, showing remarkable training efficiency despite the small dataset.
  • Superlet Performance: Contrary to expectations, Superlet transforms generally underperformed compared to standard Mel-spectrograms, yielding lower AUC scores (0.74-0.78).

Conclusion

This research highlights that "state-of-the-art" status on large benchmarks does not automatically translate to clinical utility. In data-scarce environments, the interaction between pre-training data and input representation is critical. While CNNs remain robust competitors, the integration of contrastive loss and targeted transfer learning can significantly enhance diagnostic accuracy. Future work will focus on expanding the sample size and stratifying models by patient demographics to ensure broader clinical robustness.

Summary updated

Abstract

This study assesses deep learning models for audio classification in a clinical setting with the constraint of small datasets reflecting real-world prospective data collection. We analyze CNNs, including DenseNet and ConvNeXt, alongside transformer models like ViT, SWIN, and AST, and compare them against pre-trained audio models such as YAMNet and VGGish. Our method highlights the benefits of pre-training on large datasets before fine-tuning on specific clinical data. We prospectively collected two first-of-their-kind patient audio datasets from stroke patients. We investigated various preprocessing techniques, finding that RGB and grayscale spectrogram transformations affect model performance differently based on the priors they learn from pre-training. Our findings indicate CNNs can match or exceed transformer models in small dataset contexts, with DenseNet-Contrastive and AST models showing notable performance. This study highlights the significance of incremental marginal gains through model selection, pre-training, and preprocessing in sound classification; this offers valuable insights for clinical diagnostics that rely on audio classification.
BibTeX
@misc{mahdi2024tuning,
  author = {Hamza Mahdi and Eptehal Nashnoush and Rami Saab and Arjun Balachandar and Rishit Dagli and Lucas X. Perri and Houman Khosravani},
  title = {{Tuning In: Analysis of Audio Classifier Performance in Clinical Settings with Limited Data}},
  year = {2024},
  eprint = {2402.10100},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.10100},
}
Figure