Abstract
Self-supervised learning (SSL) has become a cornerstone of modern speech processing. However, standard models often exhibit a hierarchical "division of labor," where lower layers focus on speaker identity and upper layers on phonetic content. We propose Intermediate Layer Supervision (ILS-SSL), which applies self-supervised losses to middle layers of the Transformer stack. This approach forces the model to prioritize phonetic content earlier, specializing the architecture for Automatic Speech Recognition (ASR). Our experiments demonstrate that ILS-SSL achieves up to a 25% relative reduction in Word Error Rate (WER) over the HuBERT baseline, especially in low-resource scenarios.
Introduction
Current speech SSL models like wav2vec 2.0 and HuBERT are designed as general-purpose feature extractors. While effective, this versatility often leads to inefficient use of network capacity for specific tasks like ASR. Previous research indicates that in deep models, only a subset of middle-to-top layers effectively captures phoneme information. ILS-SSL addresses this by "freeing up" capacity in the lower layers, compelling them to ignore speaker-related noise and focus on phonetic units from the start.
Methodology
The ILS-SSL framework builds upon the HuBERT architecture, which utilizes a CNN-based feature encoder followed by a stack of Transformer blocks.
Intermediate Supervision
The core innovation is the application of masked prediction loss at intermediate layers in addition to the final layer. For a 12-layer BASE model, an additional SSL loss is typically applied at layer 4. The targets for these losses are discrete cluster assignments (pseudo-labels) derived from previous iterations of the model.
Training Stability and Positional Encoding
To handle numerical instability in 16-bit precision (fp16) training, we implemented a softmax translation invariance trick to prevent attention score explosions. Additionally, we incorporated Bucket Relative Position Embeddings, which add a learnable bias based on the temporal distance between keys and queries, enhancing the model's ability to capture long-range dependencies in audio.
Results and Analysis
We evaluated ILS-SSL on the LibriSpeech dataset across various fine-tuning regimes (1h, 10h, 100h, and 960h).
Phonetic Content Extraction
Analysis using Phonetic Normalized Mutual Information (PNMI) and cluster purity metrics confirms that ILS-SSL forces phonetic learning into earlier layers. As shown in Figure 2, layers 2-12 in the ILS-SSL model correlate much more strongly with phonetic units compared to the HuBERT baseline, which only achieves similar correlation levels starting at layer 5.
ASR Performance
The performance gains are significant:
- BASE Model (100h): Achieved a 25.4% relative WER reduction on "test-clean" and 23.5% on "test-other" without a language model.
- Low Resource (1h): Reduced WER from 20.9 to 17.9, highlighting the value of specialized pre-trained representations when labeled data is scarce.
- LARGE Model (960h): Reached a state-of-the-art WER of 1.8/3.2 (clean/other).
The Capacity Trade-off
While ILS-SSL excels at ASR and semantic tasks, its specialization comes at a cost. On the SUPERB benchmark, the model showed decreased performance in Speaker Identification (SID), confirming that the model intentionally discards speaker-specific information to optimize for content.
Conclusion
Intermediate Layer Supervision effectively reallocates model capacity to improve speech recognition performance. By forcing lower layers to learn phonetic representations, ILS-SSL achieves substantial WER reductions. Future work will explore the joint pre-training of audio and text to further bridge the gap between acoustic and linguistic modeling.
Summary updated