Improving Self-Supervised Learning for Speech Recognition with Intermediate Layer Supervision

Chengyi Wang, Yu Wu, Sanyuan Chen, Shujie Liu, Jinyu Li, Yao Qian, Zhenglu Yang

Abstract

Self-supervised learning (SSL) has become a cornerstone of modern speech processing. However, standard models often exhibit a hierarchical "division of labor," where lower layers focus on speaker identity and upper layers on phonetic content. We propose Intermediate Layer Supervision (ILS-SSL), which applies self-supervised losses to middle layers of the Transformer stack. This approach forces the model to prioritize phonetic content earlier, specializing the architecture for Automatic Speech Recognition (ASR). Our experiments demonstrate that ILS-SSL achieves up to a 25% relative reduction in Word Error Rate (WER) over the HuBERT baseline, especially in low-resource scenarios.

Introduction

Current speech SSL models like wav2vec 2.0 and HuBERT are designed as general-purpose feature extractors. While effective, this versatility often leads to inefficient use of network capacity for specific tasks like ASR. Previous research indicates that in deep models, only a subset of middle-to-top layers effectively captures phoneme information. ILS-SSL addresses this by "freeing up" capacity in the lower layers, compelling them to ignore speaker-related noise and focus on phonetic units from the start.

Methodology

The ILS-SSL framework builds upon the HuBERT architecture, which utilizes a CNN-based feature encoder followed by a stack of Transformer blocks.

Intermediate Supervision

The core innovation is the application of masked prediction loss at intermediate layers in addition to the final layer. For a 12-layer BASE model, an additional SSL loss is typically applied at layer 4. The targets for these losses are discrete cluster assignments (pseudo-labels) derived from previous iterations of the model.

Figure 1: Architecture of the ILS-SSL model showing intermediate and final layer supervision.

Training Stability and Positional Encoding

To handle numerical instability in 16-bit precision (fp16) training, we implemented a softmax translation invariance trick to prevent attention score explosions. Additionally, we incorporated Bucket Relative Position Embeddings, which add a learnable bias based on the temporal distance between keys and queries, enhancing the model's ability to capture long-range dependencies in audio.

Results and Analysis

We evaluated ILS-SSL on the LibriSpeech dataset across various fine-tuning regimes (1h, 10h, 100h, and 960h).

Phonetic Content Extraction

Analysis using Phonetic Normalized Mutual Information (PNMI) and cluster purity metrics confirms that ILS-SSL forces phonetic learning into earlier layers. As shown in Figure 2, layers 2-12 in the ILS-SSL model correlate much more strongly with phonetic units compared to the HuBERT baseline, which only achieves similar correlation levels starting at layer 5.

Figure 2: Comparison of phonetic information capture (PNMI and purity) across layers for HuBERT and ILS-SSL.

ASR Performance

The performance gains are significant:

  • BASE Model (100h): Achieved a 25.4% relative WER reduction on "test-clean" and 23.5% on "test-other" without a language model.
  • Low Resource (1h): Reduced WER from 20.9 to 17.9, highlighting the value of specialized pre-trained representations when labeled data is scarce.
  • LARGE Model (960h): Reached a state-of-the-art WER of 1.8/3.2 (clean/other).

The Capacity Trade-off

While ILS-SSL excels at ASR and semantic tasks, its specialization comes at a cost. On the SUPERB benchmark, the model showed decreased performance in Speaker Identification (SID), confirming that the model intentionally discards speaker-specific information to optimize for content.

Conclusion

Intermediate Layer Supervision effectively reallocates model capacity to improve speech recognition performance. By forcing lower layers to learn phonetic representations, ILS-SSL achieves substantial WER reductions. Future work will explore the joint pre-training of audio and text to further bridge the gap between acoustic and linguistic modeling.

Summary updated

Abstract

Recently, pioneer work finds that self-supervised pre-training methods can improve multiple downstream speech tasks, because the model utilizes bottom layers to learn speaker-related information and top layers to encode content-related information. Since the network capacity is limited, we believe the speech recognition performance could be further improved if the model is dedicated to audio content information learning. To this end, we propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL), which forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly, which achieves a 23.5%/11.6% relative word error rate reduction in the w/o language model setting for Base/Large models. Detailed analysis shows the bottom layers of our model have a better correlation with phonetic units, which is consistent with our intuition and explains the success of our method for ASR. We will release our code and model at https://github.com/microsoft/UniSpeech.
BibTeX
@inproceedings{wang2022improving,
  author = {Chengyi Wang and Yu Wu and Sanyuan Chen and Shujie Liu and Jinyu Li and Yao Qian and Zhenglu Yang},
  title = {{Improving Self-Supervised Learning for Speech Recognition with Intermediate Layer Supervision}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747022},
}

Loading related content...

Figure