Introduction
Modern end-to-end (E2E) automatic speech recognition (ASR) models have significantly simplified the speech processing pipeline. However, their performance is heavily dependent on large-scale transcribed datasets. In low-resource scenarios, manual transcription is a major bottleneck, often requiring ten times the audio duration to produce accurate labels.
Active Learning (AL) offers a solution by strategically selecting the most informative samples for human annotation. While standard uncertainty-based sampling (Least Confidence) is effective, it often suffers from sampling bias, where the model over-represents specific speakers or acoustic conditions. This paper proposes a Regularized Active Learning approach that balances model uncertainty with speaker diversity to improve generalization.
Methodology: Regularized Least Confidence (RLC)
The core contribution is the Regularized Least Confidence (RLC) sampling method. This hybrid approach scores unlabeled samples based on two primary factors:
- Uncertainty (Least Confidence): The model estimates its own uncertainty by calculating the path probability of the most likely transcript during decoding. A lower probability indicates higher uncertainty.
- Diversity (i-vectors): To ensure a representative training set, the authors utilize i-vectors—unsupervised speaker embeddings—to cluster the audio pool. A diversity score is then applied to favor samples from under-represented clusters.
The final selection criterion is a weighted combination of these scores, controlled by a hyperparameter ($\lambda$). This regularization prevents the model from hyper-focusing on a narrow subset of difficult speakers.
The validity of using uncertainty as a selection signal is supported by the high correlation (0.752) observed between the model's "least confidence" score and the actual Character Error Rate (CER), as shown in the figure above. This confirms that the model's internal confidence is a reliable proxy for potential error.
Experimental Results
The RLC method was evaluated on the LibriSpeech (English) and Corpus of Spontaneous Japanese (CSJ) datasets. In both cases, RLC consistently outperformed random sampling and standard Least Confidence (LC) sampling.
Performance on LibriSpeech
On the LibriSpeech dataset, starting from a 20-hour seed, the RLC method achieved substantial gains. As shown in the table below, RLC maintains a performance edge even as the labeled pool grows to 140 hours.
Specifically, at the 20-hour mark, RLC provided a 9.8% relative improvement in Word Error Rate (WER) and a 10% improvement in CER compared to random selection. The results suggest that active learning remains effective even as the volume of training data increases.
Performance on Japanese (CSJ)
The results were even more pronounced for the Japanese dataset. For the 20-hour sampling task, the regularized approach yielded approximately a 12.5% relative improvement in CER over random sampling, highlighting its effectiveness in diverse linguistic contexts and spontaneous speech.
Conclusion and Future Work
Regularized Active Learning effectively addresses the data scarcity problem in E2E ASR by combining model uncertainty with unsupervised speaker diversity. By using i-vectors as a regularizer, the system avoids the pitfalls of sampling bias common in traditional active learning.
While the computational cost of decoding the unlabeled pool remains a challenge, the significant reduction in required transcription hours makes RLC a compelling choice for low-resource speech recognition. Future research may explore more modern embeddings, such as x-vectors, and automated methods for dynamic hyperparameter tuning.
Summary updated