PMIndia - A Collection of Parallel Corpora of Languages of India

Barry Haddow, Faheem KirefuarXiv 2020 · OpenReview

Abstract

Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56,000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.

Introduction

Despite having hundreds of millions of speakers, many South Asian languages remain "under-resourced" in Natural Language Processing (NLP). High-quality Machine Translation (MT) systems require large parallel corpora—collections of sentences paired with their translations. Existing datasets for Indian languages are often restricted to narrow domains like religious texts or technical manuals, which do not generalize well to everyday news or conversation. To address this, researchers identified the website of the Prime Minister of India (pmindia.gov.in) as a source of high-quality, multi-lingual news updates.

Methodology

The PMIndia corpus was constructed using a custom pipeline designed to handle the unique challenges of Indic scripts:

  1. Crawling and Extraction: A scraper was built to navigate the site's "infinite scroll" architecture, using the Alcazar toolkit to extract text while filtering out noise like embedded tweets.
  2. Preprocessing: The Moses sentence splitter was extended to handle Indian punctuation and non-breaking prefixes (e.g., titles like "Mr.") to prevent incorrect sentence breaks.
  3. Sentence Alignment: The team compared two alignment methods: hunalign (a traditional length-based and dictionary-based approach) and Vecalign (a modern approach using LASER embeddings).
  4. Validation: To ensure high data quality, the final corpus focuses on the intersection where both alignment methods agree.

Dataset Characteristics

The resulting PMIndia corpus pairs English with 13 major Indian languages: Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Manipuri, Marathi, Odia, Punjabi, Tamil, Telugu, and Urdu. The dataset is released under a permissive CC-BY-4.0 license.

Number of sentence pairs extracted for each language pair using different alignment methods. The intersection of hunalign and Vecalign was used for the final corpus.

As shown in the table above, the scale of the dataset varies by language pair, with the Hindi-English pair containing over 56,000 sentences in the intersection set. This represents a significant increase in available high-quality parallel data for these languages.

Experimental Results

Neural Machine Translation (NMT) models were trained to evaluate the utility of the corpus. The results highlight significant differences in performance across language families and translation directions.

BLEU scores for NMT models trained on the PMIndia corpus, showing performance for translation into English (X-en) and from English (en-X).

Key findings include:

  • Hindi-English Performance: This pair achieved the highest scores, with a BLEU score of 23.4 for English-to-Hindi.
  • Dravidian Language Challenges: Languages like Malayalam (ml) and Tamil (ta) showed much lower scores (e.g., 1.8 for en-ml), reflecting the linguistic complexity of these agglutinative languages and the relative scarcity of data.
  • Directionality: Translation into English (X-en) consistently outperformed translation from English (en-X), likely due to the model's familiarity with English structures.
  • Alignment Quality: Human evaluation confirmed that when both alignment algorithms agreed, the accuracy reached 94% for the Hindi-English pair.

Discussion and Conclusion

The analysis revealed an "indirect translation" effect: articles are often written in Hindi first, then translated into English and other regional languages. This can lead to looser alignments for non-Hindi pairs. Furthermore, human evaluation indicated that while modern embedding-based methods are powerful, traditional heuristics like hunalign remain highly effective for specific language pairs like English-Tamil.

The PMIndia corpus provides a foundational resource for Indian language NLP. Future work will focus on creating a multi-parallel version to enable direct translation between Indian languages without English as a pivot, and continuing to expand the dataset as more content is published on the source website.

Summary updated

Abstract

Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.
BibTeX
@misc{haddow2020pmindia,
  author = {Barry Haddow and Faheem Kirefu},
  title = {{PMIndia - A Collection of Parallel Corpora of Languages of India}},
  year = {2020},
  eprint = {2001.09907},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2001.09907},
}
Figure