Abstract
Self-training (ST) and self-supervised learning (SSL) methods have demonstrated strong improvements in automatic speech recognition (ASR). In spite of these advances, to the best of our knowledge, there is no analysis of how the composition of the labelled and unlabelled datasets used in these methods affects the results. In this work we aim to analyse the effect of number of speakers in the training data on a recent SSL algorithm (wav2vec 2.0), and a recent ST algorithm (slimIPL). We perform a systematic analysis on both labeled and unlabeled data by varying the number of speakers while keeping the number of hours fixed and vice versa. Our findings suggest that SSL requires a large amount of unlabeled data to produce high accuracy results, while ST requires a sufficient number of speakers in the labelled data, especially in the low-regime setting. In this manner these two approaches improve supervised learning in different regimes of data composition.
Introduction
In the quest to build better ASR systems, the field has shifted from purely supervised learning to semi-supervised methods like Self-Supervised Learning (SSL) (e.g., wav2vec 2.0) and Self-Training (ST) (e.g., slimIPL). These methods allow models to learn from vast amounts of unlabeled audio. However, data collection is expensive. If you have a fixed budget, should you record one person for 100 hours or 100 people for one hour each? While we know "more data is better," the specific impact of data composition—the balance between total hours and the number of unique speakers—has remained under-explored, particularly for low-resource settings.
Methodology
The researchers conducted a systematic "grid search" on data composition using two state-of-the-art algorithms: wav2vec 2.0 (SSL) and slimIPL (ST). Using subsets of the LibriSpeech dataset, they isolated the effects of speakers versus hours by carefully sampling labeled sets (from train-clean-100) and unlabeled sets (from train-clean-360 and train-other-500). This allowed them to hold the number of hours constant while varying the number of speakers, and vice versa, evaluating performance via Word Error Rate (WER).
Key Insights and Results
The Impact of Speaker Diversity
A primary finding is that Self-Training is "Diversity-Hungry." In low-resource settings, such as having only 10 hours of labeled data, increasing the number of speakers significantly improves performance. For slimIPL, moving from 24 to 120 speakers reduced WER from 28% to 23% when paired with 100 hours of unlabeled data.
Conversely, SSL is "Volume-Hungry." For models like wav2vec 2.0, the total volume of unlabeled hours is the dominant factor. For instance, wav2vec 2.0 Large saw WER drop from 49% to 23% simply by increasing unlabeled data from 100 to 400 hours, while labeled speaker diversity offered only marginal gains.
The Speaker Plateau
Performance gains from adding speakers are not infinite. The study observed a plateau after approximately 120 to 185 speakers. Beyond this threshold, the returns on adding more unique voices diminish compared to the benefits of increasing total speech hours.
Practical Recommendations
Based on these findings, the authors propose specific data composition targets to optimize ASR performance. For SSL, the focus should be on amassing over 500 hours of unlabeled data. For ST, ensuring a labeled set with at least 120 speakers is crucial for stability and accuracy.
Conclusion
This analysis reveals that SSL and ST have complementary strengths: SSL extracts maximum value from massive unlabeled volumes, while ST anchors the model using speaker diversity in the labeled set. Future work may explore if these thresholds hold across different languages or noisier, out-of-domain datasets.
Summary updated