Introduction
Can models trained exclusively on human speech "understand" the sound of a gunshot, a beehive, or a cello? While self-supervised learning (SSL) has revolutionized speech processing, its effectiveness on non-speech audio has remained largely unexplored. This paper demonstrates that self-supervised speech models like wav2vec 2.0 and HuBERT are surprisingly versatile audio feature extractors, though they possess a major blind spot for pitch. By ensembling these speech models with a dedicated pitch-estimation model (CREPE), the authors created a "holistic" representation that outperformed state-of-the-art audio models across a diverse 16-dataset benchmark.
The Challenge of General Audio Representation
In recent years, SSL models like HuBERT and wav2vec 2.0 have learned to process speech by predicting masked parts of an audio signal, effectively teaching themselves the structure of language without needing millions of human-labeled transcripts.
However, most of this research has stayed within the "speech bubble." It was previously unclear whether these models were learning general acoustic principles or just the specific quirks of human phonemes. Furthermore, while these models are powerful, they are typically used in isolation. The authors sought to address two main questions:
- How well do speech-centric SSL models generalize to non-speech tasks, such as environmental sound detection or music classification?
- Can we combine multiple SSL models to create a more holistic representation of audio that covers the weaknesses of any single model?
Methodology: An Ensemble Framework
The authors proposed an ensemble framework evaluated during the NeurIPS 2021 HEAR (Holistic Evaluation of Audio Representations) Challenge. Their approach relies on combining multiple models and aggregating their internal representations.
The framework utilizes three primary techniques to maximize feature extraction:
- Intra-model Feature Aggregation: Instead of just using the output of the very last layer of a Transformer (like wav2vec 2.0), the authors averaged the hidden states of all layers. This captures a rich mix of low-level acoustic features and high-level semantic information.
- Inter-model Feature Concatenation: They combined three distinct models to cover different acoustic domains:
- wav2vec 2.0 & HuBERT: Powerful speech SSL models.
- CREPE: A deep convolutional regressor specifically designed for pitch estimation.
- Linear Interpolation: Because these models have different output frequencies and dimensions, linear interpolation was used to align them in time before concatenating them into a single, comprehensive feature vector.
Evaluation Datasets
The study utilized the HEAR Benchmark, which consists of 19 tasks across 16 diverse datasets to test the true generalization capabilities of the models.
As shown above, the datasets are categorized into four main domains: Speech (e.g., emotion, language ID), Instruments (e.g., pitch, percussion), Environmental Sounds (e.g., beehives, gunshots), and General Audio (e.g., genre classification).
Key Insights and Discoveries
- The Pitch Blind Spot: The most striking finding was that while HuBERT and wav2vec 2.0 are masters of speech, they are nearly "tone-deaf." They performed poorly on fine-grained music tasks like pitch classification. This suggests that the self-supervised tasks used to train speech models (like predicting masked units) do not force the model to prioritize precise frequency information.
- Hidden Layers are Gold Mines: Averaging all hidden layers (Feature Aggregation) consistently outperformed using just the last layer. For example, on the DCASE 2016 task, wav2vec 2.0’s performance jumped from 66.3 to 79.8 simply by including the intermediate layers.
- Unexpected Generalization: Despite being trained only on speech, HuBERT achieved the second-best result among all single models on the Gunshot dataset. This indicates that the model accidentally learned to preserve spatial and distance information from the sound source.
- Complementary Strengths: wav2vec 2.0 and HuBERT often disagreed on which datasets they excelled at. By concatenating them, the ensemble became more robust, effectively "filling in" each other's gaps.
Results
The resulting ensemble framework, dubbed fusion_cat_xwc, proved to be a powerhouse in the HEAR Challenge:
- Top Performance: It achieved the highest scores on 8 out of 18 tasks, including Beijing Opera Percussion, CREMA-D, and Vocal Imitations.
- Beating Audio Specialists: The ensemble often outperformed models trained specifically on general audio (like PANNs or PaSST) on tasks like Speech Commands and Mridangam Stroke.
- CREPE to the Rescue: Adding the CREPE model to the ensemble dramatically improved music-related performance. On the NSynth Pitch 50h task, HuBERT alone scored 42.9% accuracy; the ensemble with CREPE reached a staggering 88.5%.
Limitations and Future Work
While highly effective, the authors noted a few significant hurdles with this approach. Running three large models (HuBERT xLarge, wav2vec 2.0 Large, and CREPE) simultaneously is computationally expensive and increases inference latency. Additionally, concatenating features results in very long vectors, requiring downstream "shallow" models to have more parameters, which might lead to overfitting on smaller datasets. Finally, even with CREPE, the models still struggled with complex, fine-grained music tasks compared to specialized music-AI architectures.
Future work could involve "distilling" this massive ensemble into a single, smaller model that retains the holistic knowledge without the prohibitive computational footprint.
Summary updated