Abstract
Self-supervised learning (SSL) leverages large datasets of unlabeled speech to reach impressive performance with reduced amounts of annotated data. The high number of proposed approaches fostered the emergence of comprehensive benchmarks that evaluate their performance on a set of downstream tasks exploring various aspects of the speech signal. However, while the number of considered tasks has been growing, most proposals rely upon a single downstream architecture that maps the frozen SSL representations to the task labels. This study examines how benchmarking results are affected by changes in the probing head architecture. Interestingly, we found that altering the downstream architecture structure leads to significant fluctuations in the performance ranking of the evaluated models. Against common practices in speech SSL benchmarking, we evaluate larger-capacity probing heads, showing their impact on performance, inference costs, generalization and multi-level feature exploitation.
Introduction
In the field of speech processing, Self-Supervised Learning (SSL) models like wav2vec 2.0 and HuBERT have become the standard for high-performance representation learning. To compare these models, the community relies on benchmarks such as SUPERB, which typically freeze the SSL "encoder" and train a very simple "probing head" (e.g., a linear layer or shallow RNN) for specific downstream tasks.
This paper argues that this "simple probe" philosophy creates two major issues. First, it leads to Ranking Instability, where a model that appears superior with a linear probe might perform poorly with a more realistic architecture. Second, it causes Complexity Transfer, forcing the SSL encoder to perform all computational heavy lifting because the probes are too weak, which encourages the development of unnecessarily massive encoders.
Methods
The researchers conducted a systematic "stress test" of current benchmarking practices by evaluating nine state-of-the-art SSL models across seven diverse speech tasks. The core of the experiment involved comparing two distinct sets of probing heads:
- First Set (Standard): Low-capacity probes similar to those used in the SUPERB benchmark.
- Second Set (Proposed): Higher-capacity, modern architectures like Conformers and ECAPA-TDNNs.
Beyond accuracy, the study measured inference cost (G-MACs), generalization to out-of-distribution (OOD) data, and how models weighted information across internal layers.
Key Results and Insights
The Ranking Flip and Correlation Collapse
Except for LibriSpeech, the performance rankings of SSL models changed significantly when the probing head was upgraded. The Spearman correlation between rankings with small versus large probes was as low as 0.15 for tasks like Basque ASR, suggesting that current benchmarks may be overfitting to simplistic decoder choices.
Performance and Efficiency
Upgrading the decoder led to relative improvements of up to 46.5% for Speaker Verification and 27.3% for Intent Classification. Crucially, while a Conformer head has more parameters than a linear layer, its computational cost (MACs) is negligible compared to the massive SSL encoder. Upgrading the probe provides a high return on investment for performance without significantly impacting total inference cost.
Generalization and Feature Exploitation
Contrary to the common fear that larger decoders overfit, models with larger probing heads actually performed better on zero-shot, out-of-distribution tasks. Furthermore, larger probes are better at "reaching back" into the earlier layers of the SSL encoder, capturing both acoustic and linguistic nuances that simple probes tend to ignore.
Conclusion
This study demonstrates that the choice of probing head is a critical, yet often overlooked, variable in SSL benchmarking. By switching to higher-capacity probing heads, researchers found that smaller SSL models can often outperform larger ones and generalize more effectively. The authors suggest that benchmarks should move toward "unconstrained" tracks and have released their code as the MP3S (Multi-Probe Speech Self-Supervision) sub-library within SpeechBrain to encourage more robust evaluation practices.
Summary updated