TL;DR
This paper introduces a "Gradient Mask" technique to improve pseudo-labeling for end-to-end speech recognition. By selectively masking parts of the input during training on pseudo-labeled data, the model learns more robust acoustic representations and becomes less susceptible to noisy labels, achieving competitive results with fewer resources.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) models typically require large amounts of transcribed audio data for training. Acquiring such labeled data is expensive and time-consuming. Semi-supervised learning, particularly pseudo-labeling, offers a solution by leveraging abundant unlabeled data. In pseudo-labeling, a model trained on a small labeled dataset generates "pseudo-labels" for unlabeled data, which are then used to retrain the model. However, this process is sensitive to the quality of the pseudo-labels; noisy or incorrect labels can degrade model performance, especially in end-to-end ASR systems. Existing methods often rely on complex confidence measures or data filtering to mitigate this, which can be imperfect. This paper aims to improve the robustness and effectiveness of pseudo-labeling by integrating ideas from self-supervised learning, specifically masked prediction.
Methods
The core innovation is the Gradient Mask technique. It's applied during the pseudo-labeling phase of training. Here's how it works:
- Seed Model Training: A standard end-to-end ASR model (specifically, a Transducer model in their experiments) is first trained on a small labeled dataset.
- Pseudo-Label Generation: This seed model is then used to generate pseudo-labels for a larger unlabeled dataset.
- Gradient Mask Application: When training a "student" model on this combined dataset (labeled + pseudo-labeled), the Gradient Mask is applied to the pseudo-labeled data.
- Masking Input: A random mask is generated for the input speech features. Portions of the input are replaced with a learned mask embedding.
- Masking Gradients: Crucially, during backpropagation, gradients are only allowed to flow through the encoder for the masked parts of the input. Gradients from the unmasked parts are blocked. This forces the encoder to learn strong acoustic representations by inferring masked information from context.
- Predictor Masking: Gradients are also blocked from flowing into the prediction network when processing pseudo-labeled data, further isolating the encoder's learning.
- Joint Training: The model is trained by minimizing the standard ASR loss on both labeled and pseudo-labeled data. The Gradient Mask is not applied to the labeled data.
This approach avoids extra loss functions or complex data filtering, relying solely on modifying the gradient flow during training.
Datasets
- LibriSpeech: The experiments were conducted on the LibriSpeech dataset.
- Labeled Data: 100 hours subset (train-clean-100).
- Unlabeled Data: The remaining 860 hours (train-clean-360, train-other-500).
- Validation/Test Sets: dev-clean, dev-other, test-clean, test-other.
Key Insights and Ideas
- Masked Prediction for Robustness: The paper successfully adapts the "masked prediction" paradigm from NLP (like BERT) to improve pseudo-labeling in ASR. The intuition is that forcing the model to reconstruct or infer masked acoustic information from context leads to better, more robust representations.
- Gradient Masking as a Simple Yet Effective Regularizer: Instead of complex filtering, selectively blocking gradients during backpropagation on noisy pseudo-labels acts as a powerful regularizer. It prevents the model from being overly influenced by potentially incorrect labels.
- Encoder-Centric Learning: The Gradient Mask primarily targets the encoder, pushing it to learn better acoustic features. This is beneficial because a strong encoder is fundamental for any downstream ASR task.
- Efficiency: The method is computationally efficient, requiring no additional parameters or complex training pipelines beyond standard pseudo-labeling with a modification to gradient flow.
Key Results
- Improved WER: The proposed Gradient Mask method (GM-Iter1 and GM-Iter5) significantly outperforms standard pseudo-labeling (NST-iter1) and other semi-supervised approaches on the Librispeech 100-hour dataset.
- Competitive Performance: GM-Iter5 achieved a Word Error Rate (WER) of 4.1/8.8 for dev-clean/dev-other and 4.3/8.9 for test-clean/test-other, which is highly competitive.
- Robustness to Label Noise: An ablation study (Table 4) demonstrated that the Gradient Mask method is significantly more robust to pseudo-labels of varying quality (higher WER) compared to standard training. It even showed better performance than standard training when using noisy pseudo-labels.
- Efficiency: The authors highlight that their method achieves competitive results with fewer computational resources and smaller model sizes compared to other approaches.
Limitations and Future Work
- Limitations:
- The paper focuses on the Transducer model. While they state it's adaptable, direct experiments on other architectures like CTC or LAS are not presented.
- The optimal mask parameter (p and m) and the ratio of labeled to pseudo-labeled data (1:9) were found empirically and might require tuning for different datasets or tasks.
- Future Work:
- Extend the Gradient Mask method to other end-to-end ASR systems (e.g., LAS).
- Explore its application to other sequence-to-sequence tasks, such as machine translation.
- Investigate the theoretical underpinnings of why this gradient masking strategy is so effective.
Summary updated