Abstract
Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56,000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.
Introduction
Despite having hundreds of millions of speakers, many South Asian languages remain "under-resourced" in Natural Language Processing (NLP). High-quality Machine Translation (MT) systems require large parallel corpora—collections of sentences paired with their translations. Existing datasets for Indian languages are often restricted to narrow domains like religious texts or technical manuals, which do not generalize well to everyday news or conversation. To address this, researchers identified the website of the Prime Minister of India (pmindia.gov.in) as a source of high-quality, multi-lingual news updates.
Methodology
The PMIndia corpus was constructed using a custom pipeline designed to handle the unique challenges of Indic scripts:
- Crawling and Extraction: A scraper was built to navigate the site's "infinite scroll" architecture, using the
Alcazartoolkit to extract text while filtering out noise like embedded tweets. - Preprocessing: The Moses sentence splitter was extended to handle Indian punctuation and non-breaking prefixes (e.g., titles like "Mr.") to prevent incorrect sentence breaks.
- Sentence Alignment: The team compared two alignment methods:
hunalign(a traditional length-based and dictionary-based approach) andVecalign(a modern approach using LASER embeddings). - Validation: To ensure high data quality, the final corpus focuses on the intersection where both alignment methods agree.
Dataset Characteristics
The resulting PMIndia corpus pairs English with 13 major Indian languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Manipuri, Marathi, Odia, Punjabi, Tamil, Telugu, and Urdu. The dataset is released under a permissive CC-BY-4.0 license.
As shown in the table above, the scale of the dataset varies by language pair, with the Hindi-English pair containing over 56,000 sentences in the intersection set. This represents a significant increase in available high-quality parallel data for these languages.
Experimental Results
Neural Machine Translation (NMT) models were trained to evaluate the utility of the corpus. The results highlight significant differences in performance across language families and translation directions.
Key findings include:
- Hindi-English Performance: This pair achieved the highest scores, with a BLEU score of 23.4 for English-to-Hindi.
- Dravidian Language Challenges: Languages like Malayalam (ml) and Tamil (ta) showed much lower scores (e.g., 1.8 for en-ml), reflecting the linguistic complexity of these agglutinative languages and the relative scarcity of data.
- Directionality: Translation into English (X-en) consistently outperformed translation from English (en-X), likely due to the model's familiarity with English structures.
- Alignment Quality: Human evaluation confirmed that when both alignment algorithms agreed, the accuracy reached 94% for the Hindi-English pair.
Discussion and Conclusion
The analysis revealed an "indirect translation" effect: articles are often written in Hindi first, then translated into English and other regional languages. This can lead to looser alignments for non-Hindi pairs. Furthermore, human evaluation indicated that while modern embedding-based methods are powerful, traditional heuristics like hunalign remain highly effective for specific language pairs like English-Tamil.
The PMIndia corpus provides a foundational resource for Indian language NLP. Future work will focus on creating a multi-parallel version to enable direct translation between Indian languages without English as a pivot, and continuing to expand the dataset as more content is published on the source website.
Summary updated