The Efficacy of Self-Supervised Speech Models for Audio Representations

Tung-Yu Wu, Tsu-Yuan Hsu, Chen-An Li, Tzu-Han Lin, Hung-yi Lee

Introduction

Can models trained exclusively on human speech "understand" the sound of a gunshot, a beehive, or a cello? While self-supervised learning (SSL) has revolutionized speech processing, its effectiveness on non-speech audio has remained largely unexplored. This paper demonstrates that self-supervised speech models like wav2vec 2.0 and HuBERT are surprisingly versatile audio feature extractors, though they possess a major blind spot for pitch. By ensembling these speech models with a dedicated pitch-estimation model (CREPE), the authors created a "holistic" representation that outperformed state-of-the-art audio models across a diverse 16-dataset benchmark.

The Challenge of General Audio Representation

In recent years, SSL models like HuBERT and wav2vec 2.0 have learned to process speech by predicting masked parts of an audio signal, effectively teaching themselves the structure of language without needing millions of human-labeled transcripts.

However, most of this research has stayed within the "speech bubble." It was previously unclear whether these models were learning general acoustic principles or just the specific quirks of human phonemes. Furthermore, while these models are powerful, they are typically used in isolation. The authors sought to address two main questions:

  1. How well do speech-centric SSL models generalize to non-speech tasks, such as environmental sound detection or music classification?
  2. Can we combine multiple SSL models to create a more holistic representation of audio that covers the weaknesses of any single model?

Methodology: An Ensemble Framework

The authors proposed an ensemble framework evaluated during the NeurIPS 2021 HEAR (Holistic Evaluation of Audio Representations) Challenge. Their approach relies on combining multiple models and aggregating their internal representations.

Ensemble Framework Architecture

The framework utilizes three primary techniques to maximize feature extraction:

  • Intra-model Feature Aggregation: Instead of just using the output of the very last layer of a Transformer (like wav2vec 2.0), the authors averaged the hidden states of all layers. This captures a rich mix of low-level acoustic features and high-level semantic information.
  • Inter-model Feature Concatenation: They combined three distinct models to cover different acoustic domains:
  • wav2vec 2.0 & HuBERT: Powerful speech SSL models.
  • CREPE: A deep convolutional regressor specifically designed for pitch estimation.
  • Linear Interpolation: Because these models have different output frequencies and dimensions, linear interpolation was used to align them in time before concatenating them into a single, comprehensive feature vector.

Evaluation Datasets

The study utilized the HEAR Benchmark, which consists of 19 tasks across 16 diverse datasets to test the true generalization capabilities of the models.

HEAR Benchmark Datasets

As shown above, the datasets are categorized into four main domains: Speech (e.g., emotion, language ID), Instruments (e.g., pitch, percussion), Environmental Sounds (e.g., beehives, gunshots), and General Audio (e.g., genre classification).

Key Insights and Discoveries

  • The Pitch Blind Spot: The most striking finding was that while HuBERT and wav2vec 2.0 are masters of speech, they are nearly "tone-deaf." They performed poorly on fine-grained music tasks like pitch classification. This suggests that the self-supervised tasks used to train speech models (like predicting masked units) do not force the model to prioritize precise frequency information.
  • Hidden Layers are Gold Mines: Averaging all hidden layers (Feature Aggregation) consistently outperformed using just the last layer. For example, on the DCASE 2016 task, wav2vec 2.0’s performance jumped from 66.3 to 79.8 simply by including the intermediate layers.
  • Unexpected Generalization: Despite being trained only on speech, HuBERT achieved the second-best result among all single models on the Gunshot dataset. This indicates that the model accidentally learned to preserve spatial and distance information from the sound source.
  • Complementary Strengths: wav2vec 2.0 and HuBERT often disagreed on which datasets they excelled at. By concatenating them, the ensemble became more robust, effectively "filling in" each other's gaps.

Results

The resulting ensemble framework, dubbed fusion_cat_xwc, proved to be a powerhouse in the HEAR Challenge:

  • Top Performance: It achieved the highest scores on 8 out of 18 tasks, including Beijing Opera Percussion, CREMA-D, and Vocal Imitations.
  • Beating Audio Specialists: The ensemble often outperformed models trained specifically on general audio (like PANNs or PaSST) on tasks like Speech Commands and Mridangam Stroke.
  • CREPE to the Rescue: Adding the CREPE model to the ensemble dramatically improved music-related performance. On the NSynth Pitch 50h task, HuBERT alone scored 42.9% accuracy; the ensemble with CREPE reached a staggering 88.5%.

Limitations and Future Work

While highly effective, the authors noted a few significant hurdles with this approach. Running three large models (HuBERT xLarge, wav2vec 2.0 Large, and CREPE) simultaneously is computationally expensive and increases inference latency. Additionally, concatenating features results in very long vectors, requiring downstream "shallow" models to have more parameters, which might lead to overfitting on smaller datasets. Finally, even with CREPE, the models still struggled with complex, fine-grained music tasks compared to specialized music-AI architectures.

Future work could involve "distilling" this massive ensemble into a single, smaller model that retains the holistic knowledge without the prohibitive computational footprint.

Summary updated

Abstract

Self-supervised learning (SSL) speech models, which can serve as powerful upstream models to extract meaningful speech representations, have achieved unprecedented success in speech representation learning. However, their effectiveness on non-speech datasets is relatively less explored. In this work, we propose an ensemble framework, with a combination of ensemble techniques, to fuse SSL speech models' embeddings. Extensive experiments on speech and non-speech audio datasets are conducted to investigate the representation abilities of our ensemble method and its single constituent model. Ablation studies are carried out to evaluate the performances of different ensemble techniques, such as feature averaging and concatenation. All experiments are conducted during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline provided by competition officials. Results demonstrate SSL speech models' strong abilities on various non-speech tasks, while we also note that they fail to deal with fine-grained music tasks, such as pitch classification and note onset detection. In addition, feature ensemble is shown to have great potential on producing more holistic representations, as our proposed framework generally surpasses state-of-the-art SSL speech/audio models and has superior performance on various datasets compared with other teams in HEAR Challenge. Our code is available at https://github.com/tony10101105/HEAR-2021-NeurIPS-Challenge—NTU-GURA.
BibTeX
@inproceedings{wu2021efficacy,
  author = {Tung-Yu Wu and Tsu-Yuan Hsu and Chen-An Li and Tzu-Han Lin and Hung-yi Lee},
  title = {{The Efficacy of Self-Supervised Speech Models for Audio Representations}},
  booktitle = {HEAR @ NeurIPS},
  year = {2021},
  url = {https://openreview.net/forum?id=RyucnkBONl},
}
Figure