Investigating the Emergent Audio Classification Ability of ASR Foundation Models

Rao Ma, Adian Liusie, Mark J. F. Gales, Kate M. KnillNAACL 2024 · DOI

TL;DR

This paper explores the surprising ability of large-scale Automatic Speech Recognition (ASR) models, like Whisper, to perform audio classification tasks they were never explicitly trained for, using simple text prompts. They demonstrate that these "foundation models" can achieve competitive zero-shot performance, especially when their output probabilities are calibrated to reduce inherent biases.

What Problem Does This Paper Address?

The field of AI has seen tremendous success with "foundation models" – large, pre-trained models that can be adapted to a wide range of tasks with minimal or no task-specific training (zero-shot learning). While this has been well-established for text (LLMs) and vision-language models, the zero-shot capabilities of ASR foundation models for tasks beyond speech recognition have been largely unexplored. Existing work either focuses on tasks very similar to ASR or requires task-specific fine-tuning. This paper aims to bridge this gap by investigating how effectively ASR foundation models can perform general audio classification tasks in a zero-shot manner, without any parameter updates.

Methods

The core of the paper's methodology revolves around zero-shot prompting of ASR foundation models.

  • Prompting: The researchers use simple text templates (e.g., "This is a sound of class_label.") that are fed to the ASR model's decoder. The model then generates the probability of decoding these label sequences given an audio input.
  • Probability Calculation: The likelihood of decoding a class-specific label sequence is used as a proxy for the audio's classification. This probability is then normalized across all possible classes to obtain a probability distribution.
  • Task Calibration: To address inherent biases in the models' predictions (e.g., favoring common words from pre-training), two calibration techniques are employed:
  • Prior Matching: This method reweights the output probabilities using unsupervised data to ensure the output distribution matches the expected true prior distribution of classes.
  • Null-Input Calibration: A data-free approach that uses a "null input" (e.g., silence or synthetic noise) to estimate and correct model biases.

Datasets

The study evaluates performance on a diverse set of 8 audio classification datasets, covering 6 distinct tasks:

  • Sound Event Classification (SEC): ESC50, UrbanSound8K
  • Acoustic Scene Classification (ASC): TUT2017
  • Vocal Sound Classification (VSC): Vocal Sound
  • Emotion Recognition (ER): RAVDESS, CREMA-D
  • Music Genre Classification (MGC): GTZAN
  • Speaker Counting (SC): LibriCount

These datasets vary in size, duration, and class balance.

Key Insights and Ideas

  • Emergent Abilities in ASR: The most striking insight is that ASR foundation models, trained primarily for speech-to-text, possess significant emergent abilities for general audio classification. They can understand and categorize sounds they haven't been explicitly trained on.
  • Prompting is Key: Simple, well-designed text prompts are surprisingly effective at unlocking these zero-shot capabilities. Natural language prompts often perform better than simple class labels.
  • Calibration is Crucial: Raw model outputs are often biased, leading to skewed predictions. Calibration techniques, particularly prior matching, are essential for debiasing and significantly improving performance.
  • Model Scale Matters: Larger models generally exhibit better zero-shot performance, suggesting that scaling up ASR foundation models further enhances their cross-task generalization abilities. Multilingual models also tend to outperform English-only models at larger scales.
  • Pseudo-Speech Data Can Harm: Interestingly, the newly released Whisper large-v3, trained with pseudo-speech data, showed inferior performance, suggesting that the quality and nature of training data can impact emergent abilities.

Key Results

  • Whisper's Strong Performance: Whisper models, especially larger versions, achieved average zero-shot accuracies significantly better than random across the tested datasets. For instance, Whisper large-v2 achieved an average of 30% accuracy without calibration, and 48.2% with prior-matching.
  • Outperforming Baselines: The proposed zero-shot prompting approach with prior-matching outperformed existing baselines like AudioCLIP and CLAP by an average of 9.2%, even on tasks where CLAP was specifically tuned.
  • MMS Limitations: The MMS model, due to its CTC architecture and token-level output, struggled significantly with zero-shot audio classification, highlighting architectural differences in how models leverage pre-training.
  • Calibration Gains: Prior matching provided substantial performance boosts, improving average accuracy by up to 25% in some cases. Null-input calibration also offered improvements, especially in low-resource scenarios.
  • Audio Question Answering: Whisper also showed promise in zero-shot audio question answering, achieving 64.0% accuracy on the unanimous test set of the Clotho-AQA dataset.

Limitations and Future Work

  • Prior Matching Data Dependency: The prior-matching calibration method requires unlabelled in-domain data or a large test set, which might not be feasible in settings with strong class imbalances or limited data.
  • Generalization Caution: While promising, the authors caution that the system's generalization to tasks not considered in the paper should be approached with care, especially for real-world applications.
  • Prompt Sensitivity: While natural language prompts are effective, performance can still be sensitive to prompt wording, as shown by the robustness experiments. Ensembling prompts can mitigate this.
  • Future Work: The paper suggests exploring more advanced calibration techniques, investigating the impact of different model architectures, and further exploring the emergent abilities of ASR models for a wider range of audio tasks. The potential for using these models in low-resource scenarios is also highlighted.

Summary updated

BibTeX
@inproceedings{ma2024investigating,
  author = {Rao Ma and Adian Liusie and Mark J. F. Gales and Kate M. Knill},
  title = {{Investigating the Emergent Audio Classification Ability of ASR Foundation Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.266},
}
Figure