Towards Building ASR Systems for the Next Billion Users

Tahir Javed, Sumanth Doddapaneni, Abhigyan Raman, Kaushal Santosh Bhogale, Gowtham Ramesh, Anoop Kunchukuttan, Pratyush Kumar, Mitesh M. Khapra

Abstract

Automatic Speech Recognition (ASR) technology has seen significant advancements in high-resource languages like English, yet it remains inaccessible for the "next billion users" in the Indian subcontinent. This paper presents a large-scale effort to bridge this gap by curating a 17,000-hour speech dataset across 40 Indian languages and developing IndicWav2Vec. By utilizing self-supervised pretraining and script unification, the researchers achieved state-of-the-art results across multiple benchmarks, demonstrating the efficacy of multilingual models in low-resource settings.

Introduction

While voice-activated assistants are commonplace in the West, they often fail to understand the diverse linguistic landscape of India. Home to over 2 billion people and hundreds of languages, the region presents unique challenges for ASR: extreme data scarcity for labeled audio, complex inflectional morphology, and a wide array of scripts. This work moves away from traditional supervised learning toward a scalable self-supervised strategy to democratize speech technology.

Methodology

The researchers employed the wav2vec 2.0 framework to learn speech representations from unlabeled audio. The process involved:

  1. Multilingual Pretraining: Training "BASE" (95M parameters) and "LARGE" (317M parameters) models on over 17,000 hours of audio. Temperature-based sampling was used to balance the representation of low-resource languages.
  2. Script Unification: To facilitate cross-lingual knowledge transfer, various Indic scripts were mapped to a single Devanagari representation.
  3. Decoding: A beam search decoder integrated the acoustic model with a 6-gram language model and task-specific lexicons to refine word-level accuracy.

Dataset Curation

The study introduced a massive corpus of 17,314 hours of raw audio spanning 40 languages. This data was sourced from diverse public domains, including educational content on YouTube and public radio broadcasts from NewsOnAir.

Distribution of raw audio data across 40 Indian languages from NewsOnAir (NoA) and YouTube (YT).

Results and Discussion

The IndicWav2Vec models established new state-of-the-art (SOTA) benchmarks across three major public datasets: MSR, MUCS 2021, and OpenSLR.

Word Error Rate (WER) results across different model configurations (M0-M7) on MSR, MUCS, and OpenSLR benchmarks. Lower is better.

Key Findings

  • Pretraining Impact: Models trained from scratch (M0) performed poorly compared to those with self-supervised pretraining (M1-M3). Pretraining provides a critical "warm start" for low-resource languages.
  • Zero-Shot Transfer: The model showed remarkable performance on Sinhala, a language not included in the pretraining set, indicating that the diverse multilingual pretraining helped learn a generalized phonetic space.
  • Lexicon Importance: In low-resource scenarios, the quality of the lexicon significantly impacts Word Error Rate (WER). Using an augmented lexicon (M6) consistently improved results over standard language models.
  • Complexity of Indic Languages: Unlike English, where minimal labeled data can suffice, Indic languages require more substantial fine-tuning due to their phonetic and morphological richness.

Conclusion

IndicWav2Vec represents a significant step toward inclusive ASR for the Indian subcontinent. While the achieved SOTA results are promising, challenges such as code-mixing (e.g., "Hinglish") and the need for even larger, more diverse corpora remain. Future research will focus on improving language models and expanding the linguistic coverage to further lower the barriers for the next billion users.

Summary updated

Abstract

Recent methods in speech and language technology pretrain very LARGE models which are fine-tuned for specific tasks. However, the benefits of such LARGE models are often limited to a few resource rich languages of the world. In this work, we make multiple contributions towards building ASR systems for low resource languages from the Indian subcontinent. First, we curate 17,000 hours of raw speech data for 40 Indian languages from a wide variety of domains including education, news, technology, and finance. Second, using this raw speech data we pretrain several variants of wav2vec style models for 40 Indian languages. Third, we analyze the pretrained models to find key features: codebook vectors of similar sounding phonemes are shared across languages, representations across layers are discriminative of the language family, and attention heads often pay attention within small local windows. Fourth, we fine-tune this model for downstream ASR for 9 languages and obtain state-of-the-art results on 3 public datasets, including on very low-resource languages such as Sinhala and Nepali. Our work establishes that multilingual pretraining is an effective strategy for building ASR systems for the linguistically diverse speakers of the Indian subcontinent. Our code, data and models are available publicly at https://indicnlp.ai4bharat.org/indicwav2vec/ and we hope they will help advance research in ASR for Indic languages.
BibTeX
@inproceedings{javed2022towards,
  author = {Tahir Javed and Sumanth Doddapaneni and Abhigyan Raman and Kaushal Santosh Bhogale and Gowtham Ramesh and Anoop Kunchukuttan and Pratyush Kumar and Mitesh M. Khapra},
  title = {{Towards Building ASR Systems for the Next Billion Users}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21327},
}
Figure