Empowering Low-Resource Language ASR via Large-Scale Pseudo Labeling

Kaushal Santosh Bhogale, Deovrat Mehendale, Niharika Parasa, Sathish Kumar Reddy G, Tahir Javed, Pratyush Kumar, Mitesh M. Khapra

TL;DR

The paper introduces Pratinidhi, a robust pseudo-labeling framework designed to boost Automatic Speech Recognition (ASR) performance for low-resource languages like Hindi. By enforcing consensus between multiple base models and filtering transcripts with multimodal evaluators, the authors successfully leveraged over 28,000 hours of unlabeled YouTube audio to achieve massive Word Error Rate (WER) reductions across diverse domains without degrading out-of-domain performance.

What Problem Does This Paper Address?

State-of-the-art ASR systems for languages like English are typically trained on tens of thousands of hours of high-quality labeled data. For low-resource or under-resourced languages like Hindi, such massive datasets simply do not exist.

While pseudo-labeling (using an existing ASR model to transcribe unlabeled audio and adding those pairs back into the training set) is a popular semi-supervised alternative, it faces a chicken-and-egg problem in low-resource settings: if the initial base model isn't highly accurate, the generated pseudo-labels will be noisy, leading to poor downstream performance. The authors tackle this by designing a framework that filters out low-quality pseudo-labels using model consensus and multimodal verification.

Methods

The authors propose Pratinidhi (Sanskrit for "representative"), a framework consisting of two main pillars:

  1. Pseudo-Transcribers (Consensus): Instead of relying on a single model, Pratinidhi uses multiple ASR architectures (or different decoding heads, like CTC and RNN-T from a single hybrid Conformer-L model) to transcribe the same audio segment. A text-matching score based on Levenshtein distance ensures that a transcript is only accepted if the models strongly agree on the text.
  2. Evaluators (Filtering): Once a consensus transcript is found, it must pass two strict quality checks:
  3. Confidence Score: An entropy-based confidence metric derived from the ASR model.
  4. Multimodal Similarity: Using SONAR, a language-agnostic multimodal embedding space, the framework computes the cosine similarity between the audio embedding and the text transcript embedding to ensure they actually match semantically.

Datasets

The study utilizes a mix of existing datasets and a newly introduced benchmark:

  • PN-lab (Labeled Training): A collated dataset of 2,531 hours of Hindi audio, combining VISTAAR, INDICVOICES, and SPRING-INX.
  • PN-unlab (Unlabeled Data): A massive 28,616-hour corpus of raw Hindi audio scraped from YouTube and the Dhwani dataset. After Pratinidhi's strict consensus and filtering pipeline, this was whittled down to 1,840 hours of high-quality pseudo-labeled data (PN-pseudolab).
  • INDICYT (New Benchmark): A newly curated, human-annotated 2-hour evaluation benchmark sourced from YouTube, covering 14 distinct domains (e.g., Business, Cooking, Debates, Math, Science) to test domain generalization.
  • VISTAAR (Out-of-Domain Benchmark): Used to ensure the model doesn't overfit to the conversational/YouTube style of the pseudo-labeled data.

Key Insights and Ideas

  • The Power of Multi-System Alignment: The ablation studies highlight that relying on just an RNN-T decoder or just a multimodal filter leaves too much noise in the dataset. The combination of model consensus and multimodal embedding alignment yields the cleanest data and the best downstream ASR performance.
  • Democratizing Difficult Domains: Base ASR models trained on standard datasets typically fail on highly technical or niche topics. Pseudo-labeling diverse YouTube content acts as an effective domain-adaptation technique, providing massive accuracy boosts in complex areas like education, math, and science.
  • Immunity to Style Drift: A common fear in semi-supervised learning is that tuning a model on a specific style of unlabeled data (like YouTube videos) will degrade its performance on other domains (like read speech or call center data). The authors show that large-scale pseudo-labeling acts as a regularizer, maintaining stable performance on the out-of-domain VISTAAR benchmark.

Key Results

  • Overall WER Reduction: On the new INDICYT benchmark, adding the Pratinidhi-filtered data dropped the absolute Word Error Rate (WER) from 33.3% to 24.7%.
  • Domain-Specific Triumphs: As illustrated in the radar chart below, the model achieved dramatic improvements in traditionally difficult educational domains, such as Math and Science.
Figure 1: Comparison of our model on the INDICYTbenchmark shows improvement across all domains, with significant im- provement on Education domains like Math and Science.
  • Ablation Validation: Table 3 demonstrates that the full Pratinidhi pipeline ("PN") outperforms variants that omit filtering or use only a single transcriber head.
Table 3: Effect of combining pseudo-transribers and evaluators
  • Robustness to Audio Length: The framework provided significant WER reductions across short (2-10s), medium (10-20s), and long (20-30s) audio segments, likely because the VAD-based segmentation introduced highly varied utterance lengths into the training mix.

Limitations and Future Work

While highly effective, the authors note a few areas for improvement:

  • Single Iteration: The current study only performs a single iteration of pseudo-labeling. Implementing an iterative self-training loop (where the improved model generates even better pseudo-labels in the next round) could yield further gains.
  • Hyperparameter Sensitivity: The thresholds for consensus ($\tau$) and filtering ($\rho$) were determined via manual verification on a small subset. Automating threshold selection could make the framework more adaptable to other languages.
  • Scaling to Other Languages: The authors plan to extend this framework to the other 13 Indian languages supported by the SONAR embedding space.

Summary updated

Abstract

In this study, we tackle the challenge of limited labeled data for low-resource languages in ASR, focusing on Hindi. Specifically, we explore pseudo-labeling, by proposing a generic framework combining multiple ideas from existing works. Our framework integrates multiple base models for transcription and evaluators for assessing audio-transcript pairs, resulting in robust pseudo-labeling for low resource languages. We validate our approach with a new benchmark, IndicYT, comprising diverse YouTube audio files from multiple content categories. Our findings show that augmenting pseudo labeled data from YouTube with existing training data leads to significant performance improvements on IndicYT, without affecting performance on out-of-domain benchmarks, demonstrating the efficacy of pseudo-labeled data in enhancing ASR capabilities for low-resource languages. The benchmark, code and models developed as a part of this work will be made publicly available.
BibTeX
@inproceedings{bhogale2024empowering,
  author = {Kaushal Santosh Bhogale and Deovrat Mehendale and Niharika Parasa and Sathish Kumar Reddy G and Tahir Javed and Pratyush Kumar and Mitesh M. Khapra},
  title = {{Empowering Low-Resource Language ASR via Large-Scale Pseudo Labeling}},
  booktitle = {Interspeech},
  year = {2024},
}
Figure