TL;DR
This paper describes the first-ever low-resource Automatic Speech Recognition (ASR) challenge for Indian languages, releasing 150 hours of transcribed speech data for Tamil, Telugu, and Gujarati. The challenge aimed to foster innovation in ASR for under-resourced languages, with top-performing systems demonstrating the effectiveness of multilingual TDNN architectures and transfer learning.
What Problem Does This Paper Address?
While speech technologies have advanced significantly, many of the world's languages, including numerous Indian languages, remain "low-resource." This means they lack sufficient transcribed speech data to train accurate ASR systems. India alone has over 1500 languages, with many spoken by millions, yet only a handful have functional ASR support, and their performance lags far behind high-resource languages like English. The primary bottleneck is the scarcity of transcribed speech data, and existing large datasets are often inaccessible to academic institutions due to subscription costs. This paper addresses this gap by organizing a challenge and releasing a novel dataset to stimulate research in low-resource ASR for Indian languages.
Methods
The challenge encouraged participants to develop ASR systems using only the provided data, preserving the low-resource setting. However, participants were free to innovate across the entire ASR pipeline, including using cross-lingual or multilingual models. The organizers provided baseline systems built using standard Kaldi recipes, including GMM-HMM, Karel's DNN, and TDNN (chain) models. These baselines served as benchmarks for participants. The challenge also provided pronunciation lexicons for each language, designed to facilitate phoneme sharing and minimize out-of-vocabulary issues.
Datasets
The challenge released approximately 150 hours of transcribed speech data in total, comprising 50 hours for each of the three Indian languages: Tamil, Telugu, and Gujarati. This data included phrasal and conversational speech, transcribed in the native script of each language. For the challenge, 40 hours of training data and 5 hours of test data were provided for each language. An additional 5 hours of blind test data was used for the final evaluation.
Key Insights and Ideas
- TDNNs Dominate: Time-Delay Neural Networks (TDNNs) emerged as the most effective architecture for this low-resource setting, consistently outperforming GMM-HMM and DNN baselines across all three languages.
- Multilingualism is Key: Multilingual TDNN systems, particularly those employing transfer learning, achieved the best results, highlighting the benefit of leveraging similarities between related Indian languages.
- RNNLM for Rescoring: Recurrent Neural Network Language Models (RNNLMs) proved effective for rescoring, often improving performance over traditional n-gram models.
- End-to-End Potential: While TDNNs were strong, the paper notes the promising results of end-to-end ASR systems, some using CTC-based multilingual training, suggesting a new direction for low-resource ASR.
- Unsupervised Approaches Show Promise: Innovative approaches like unsupervised acoustic unit discovery and articulatory feature extraction using pooled data also showed potential, even if they didn't always beat the top TDNN systems.
Key Results
The challenge received 109 submissions from 18 research groups. The top-performing systems significantly outperformed the TDNN baseline. For instance, the best system, "Jilebi," achieved Word Error Rates (WERs) of:
- Tamil: 13.92%
- Telugu: 14.71%
- Gujarati: 14.06%
These results represent a substantial improvement over the baseline TDNN models, which had WERs of 19.45% for Tamil, 22.61% for Telugu, and 19.76% for Gujarati. Notably, many submitted systems, even those using simpler models, explored data sharing and multilingual strategies.
Limitations and Future Work
A significant limitation acknowledged is the relatively small amount of data (50 hours per language) provided for the challenge, which still represents a low-resource setting. While the challenge successfully stimulated research, the authors hope the released data will enable further advancements. Future work could explore more sophisticated multilingual and cross-lingual learning techniques, investigate the effectiveness of end-to-end models for a wider range of Indian languages, and leverage unsupervised or semi-supervised methods to further reduce reliance on transcribed data. The paper also suggests that the released data will be made publicly available, fostering continued research in this area.
Summary updated