More Speaking or More Speakers?

Dan Berrebbi, Ronan Collobert, Navdeep Jaitly, Tatiana Likhomanenko

TL;DR

This paper investigates a fundamental trade-off in training Automatic Speech Recognition (ASR) systems: is it better to have more total hours of speech or a greater diversity of speakers? The researchers found that while Self-Supervised Learning (SSL) thrives on a high volume of unlabeled hours, Self-Training (ST) is uniquely sensitive to speaker diversity in the labeled dataset, especially when data is scarce.

What Problem Does This Paper Address?

In the quest to build better ASR systems, the field has shifted from purely supervised learning to semi-supervised methods like Self-Supervised Learning (SSL) (e.g., wav2vec 2.0) and Self-Training (ST) (e.g., slimIPL). These methods allow models to learn from vast amounts of unlabeled audio.

However, data collection is expensive. If you have a fixed budget, should you record one person for 100 hours or 100 people for one hour each? While we know "more data is better," we haven't historically understood how the composition—the balance between total hours and the number of unique speakers—affects performance. This is particularly critical for low-resource languages where every minute of labeled data is precious.

Methods

The authors conducted a systematic "grid search" on data composition. They used two state-of-the-art algorithms:

  1. wav2vec 2.0 (SSL): A model that learns speech representations from unlabeled data before being fine-tuned on a small labeled set.
  2. slimIPL (ST): A self-training algorithm that uses a teacher model to generate "pseudo-labels" for unlabeled data, which are then used to train a student model.

The core of their method involved creating custom subsets of the LibriSpeech dataset. By carefully sampling, they could hold the number of hours constant while varying the number of speakers, and vice versa. They evaluated performance using Word Error Rate (WER) across different "regimes" of data availability.

Datasets

The study relies on LibriSpeech, a standard benchmark for ASR. To isolate the effects of speakers vs. hours, they created specific subsets:

  • Labeled Sets: Derived from train-clean-100 (100 hours, 251 speakers).
  • Unlabeled Sets: Derived from train-clean-360 and train-other-500 (totaling 860 hours and 2,087 speakers).

They produced a variety of configurations, such as "10 hours from 24 speakers" vs. "10 hours from 251 speakers," to pinpoint exactly where performance gains come from.

Key Insights and Ideas

  • SSL is "Volume-Hungry": For SSL models like wav2vec 2.0, the total number of unlabeled hours is the primary driver of success. Increasing the number of speakers in the labeled fine-tuning set provides only marginal gains.
  • ST is "Diversity-Hungry": For Self-Training, speaker diversity in the labeled set is critical. In low-resource settings (e.g., only 10 hours of labeled data), having more speakers significantly lowers the WER.
  • The Magic Number (The Plateau): Performance gains from adding more speakers aren't infinite. The researchers observed a plateau after roughly 120 to 185 speakers. Beyond this point, adding more unique voices yields diminishing returns compared to simply adding more hours.
  • Complementary Strengths: The findings explain why combining SSL and ST often works so well: they improve the model in different data composition regimes. SSL extracts the most value from massive unlabeled volumes, while ST anchors the model using speaker diversity.

Key Results

The quantitative differences are striking, particularly for Self-Training:

  • Self-Training (slimIPL): With 10 hours of labeled data and 100 hours of unlabeled data, moving from 24 speakers to 120 speakers reduced the WER from 28% to 23%. With 860 hours of unlabeled data, that same increase in speaker diversity dropped WER from 19% to 15.5%.
  • SSL (wav2vec 2.0 Large): The model showed a "drastic gain" from unlabeled volume. Averaged across speaker counts, 10 hours of labeled data paired with 100 hours of unlabeled data resulted in a 49% WER, which plummeted to 23% when the unlabeled data was increased to 400 hours.
  • Proposal Summary: The authors provide a handy reference (Table 1) suggesting that for acceptable performance, one should aim for >120 speakers and >500 hours of unlabeled data for SSL, or >100 hours for ST.

Limitations and Future Work

The authors acknowledge a few boundaries to their study:

  • Domain Consistency: All data (labeled and unlabeled) came from the same general domain (LibriSpeech audiobooks). The dynamics might change if the unlabeled data is significantly "noisier" or from a different domain than the labeled data.
  • Demographics: The study focused on the number of speakers but did not explicitly control for the diversity of accents, lexical content, or specific noise conditions.
  • Language Scope: The experiments were conducted solely on English. Future work could investigate if these "speaker vs. hour" thresholds hold true for morphologically rich or tonal languages.

Summary updated

Abstract

Self-training (ST) and self-supervised learning (SSL) methods have demonstrated strong improvements in automatic speech recognition (ASR). In spite of these advances, to the best of our knowledge, there is no analysis of how the composition of the labelled and unlabelled datasets used in these methods affects the results. In this work we aim to analyse the effect of number of speakers in the training data on a recent SSL algorithm (wav2vec 2.0), and a recent ST algorithm (slimIPL). We perform a systematic analysis on both labeled and unlabeled data by varying the number of speakers while keeping the number of hours fixed and vice versa. Our findings suggest that SSL requires a large amount of unlabeled data to produce high accuracy results, while ST requires a sufficient number of speakers in the labelled data, especially in the low-regime setting. In this manner these two approaches improve supervised learning in different regimes of data composition.
BibTeX
@inproceedings{berrebbi2023more,
  author = {Dan Berrebbi and Ronan Collobert and Navdeep Jaitly and Tatiana Likhomanenko},
  title = {{More Speaking or More Speakers?}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096078},
}

Loading related content...

Figure