Abstract: Self-supervised learning (SSL) speech models, which can serve as powerful upstream models to extract meaningful speech representations, have achieved unprecedented success in speech representation learning. However, their effectiveness on non-speech datasets is relatively less explored. In this work, we propose an ensemble framework, with a combination of ensemble techniques, to fuse SSL speech models' embeddings. Extensive experiments on speech and non-speech audio datasets are conducted to investigate the representation abilities of our ensemble method and its single constituent model. Ablation studies are carried out to evaluate the performances of different ensemble techniques, such as feature averaging and concatenation. All experiments are conducted during NeurIPS 2021 HEAR Challenge as a standard evaluation pipeline provided by competition officials. Results demonstrate SSL speech models' strong abilities on various non-speech tasks, while we also note that they fail to deal with fine-grained music tasks, such as pitch classification and note onset detection. In addition, feature ensemble is shown to have great potential on producing more holistic representations, as our proposed framework generally surpasses state-of-the-art SSL speech/audio models and has superior performance on various datasets compared with other teams in HEAR Challenge.
1. Introduction
In recent years, Self-Supervised Learning (SSL) has revolutionized speech processing. Models like HuBERT and wav2vec 2.0 learn to understand audio by predicting masked segments of speech, requiring no human labels. However, most research focuses strictly on speech tasks such as transcription or speaker identification. This paper addresses whether models trained only on human voices can generalize to understand diverse sounds like gunshots, barking dogs, or musical instruments, and how to effectively fuse distinct model perspectives into a single holistic representation.
2. Methods
The researchers propose an ensemble framework, designated NTU-GURA, which utilizes two primary techniques to fuse information from multiple upstream models:
- Intra-model Feature Aggregation: Instead of relying solely on the final layer of a Transformer, the framework averages the hidden states of all layers. This captures a balanced mix of low-level acoustic features and high-level semantic information.
- Inter-model Feature Concatenation: The system combines three distinct models: wav2vec 2.0 and HuBERT (for general and speech features) and CREPE (a CNN-based model specialized for pitch). Because these models have different output sizes and temporal resolutions, linear interpolation is used to align them before concatenating their features into a unified vector.
3. Datasets and Evaluation
The study utilizes the NeurIPS 2021 HEAR (Holistic Evaluation of Audio Representations) Challenge benchmark. This massive suite is designed to test "all-purpose" audio models across 19 tasks categorized into speech, instruments, environmental sounds, and general audio.
4. Key Insights and Results
4.1 Cross-Domain Generalization
Surprisingly, speech-pretrained models like HuBERT achieved the second-best result on the Gunshot Triangulation dataset. This suggests that speech pretraining inadvertently teaches models to perceive spatial characteristics and distance in sound, even when not explicitly trained for environmental audio.
4.2 The "Tone-Deaf" Problem and Recovery
Despite their power, speech SSL models initially failed at fine-grained music tasks like pitch classification. They are essentially "tone-deaf" because speech pretraining does not require the high-frequency resolution needed for musical notes. However, adding CREPE to the ensemble drastically improved performance. While HuBERT alone achieved only 18.4% accuracy on the NSynth pitch task, the ensemble skyrocketed to 88.5%.
4.3 Benchmark Performance
The ensemble model, fusion_cat_xwc, was a top performer in the HEAR Challenge, achieving the highest scores on 8 out of 18 tasks, including Beijing Opera, CREMA-D, and Vocal Imitations. It consistently outperformed individual "Base" and "Large" versions of its constituent models, proving that the fusion strategy successfully created a more robust and holistic representation.
5. Limitations and Future Work
The primary drawback of this approach is computational cost; running three large deep learning models simultaneously is resource-intensive. Furthermore, concatenating features leads to high-dimensional vectors that may contain redundancy. Future work will explore distilling these ensembles into single, smaller models and improving performance on extremely fine-grained tasks like music note onset detection.
Summary updated