Abstract
The IndicNLPSuite is a comprehensive open-source ecosystem designed to address the resource scarcity for Indian languages in Natural Language Processing. It introduces IndicCorp, an 8.8-billion-token corpus across 11 major Indian languages; IndicFT, a set of sub-word embeddings; IndicBERT, a compact multilingual ALBERT-based model; and IndicGLUE, a standardized evaluation benchmark. Our results demonstrate that models trained on this diverse, large-scale data significantly outperform existing multilingual models like mBERT and XLM-R on regional tasks.
Introduction
Despite being spoken by over a billion people, Indian (Indic) languages have historically been classified as "low-resource" in the digital NLP landscape. Existing state-of-the-art models often rely on English-centric data or limited Wikipedia scrapes for non-English languages. These sources fail to capture the morphological richness and linguistic diversity inherent in the Indian subcontinent. Furthermore, the lack of a unified evaluation framework similar to GLUE has hindered progress in measuring model performance across varied tasks like sentiment analysis and news classification.
The IndicNLPSuite Ecosystem
To bridge this gap, the IndicNLPSuite provides a three-pronged approach to building a comprehensive NLP infrastructure:
- IndicCorp: A massive monolingual corpus built by crawling news articles, magazines, and blogs. We employed the Murmurhash algorithm for aggressive de-duplication to ensure high data quality.
- IndicFT: Word embeddings utilizing the FastText architecture. By incorporating sub-word information (n-grams), IndicFT effectively handles the complex morphological variations typical of Indic languages.
- IndicBERT: A multilingual language model based on the ALBERT (A Lite BERT) architecture. ALBERT's parameter-sharing makes it computationally efficient and suitable for real-world deployment while maintaining high performance.
IndicGLUE: A Standardized Benchmark
IndicGLUE provides a diverse set of tasks to evaluate model understanding. It includes semi-automatically created datasets for Headline Prediction and Wikipedia Section-Title Prediction, manual translations of English benchmarks (e.g., Winograd NLI), and integrated public datasets for Named Entity Recognition (NER) and Sentiment Analysis. This benchmark allows for the first-ever standardized performance baseline for several languages like Odia and Assamese.
Key Results and Insights
Our experiments reveal that the quality of embeddings and models scales directly with the size of the monolingual corpus. Languages with larger data pools, such as Hindi and Bengali, showed the most significant gains.
As shown in the results, IndicFT outperformed standard FastText models trained on Wikipedia across nearly all classification tasks, achieving an average improvement of approximately 7%. Additionally, IndicBERT consistently outperformed mBERT and XLM-R on the majority of IndicGLUE tasks. Notably, in Cross-Lingual Sentence Retrieval, IndicBERT Large achieved an average precision of 34.10, far exceeding mBERT (21.46) and XLM-R (13.74).
Conclusion and Future Work
The IndicNLPSuite represents a significant step toward leveling the playing field for Indian NLP. While currently weighted toward news content, future iterations will aim to include more conversational data and expand coverage beyond the initial 11 languages. We also plan to explore more complex natural language understanding tasks such as summarization and dialogue systems to further empower the Indic NLP community.
Summary updated