Leveraging the Cross-Domain & Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpus

Pooja Singh, Shashwat Bhardwaj, Vaibhav Sharma, Sandeep KumarEMNLP 2025

TL;DR

This paper introduces the Bhili-Hindi-English Parallel Corpus (BHEPC), the first large-scale, human-curated dataset for the underrepresented Bhili language. They use this corpus to benchmark various multilingual language models, finding that while proprietary models excel, the fine-tuned NLLB-200 model shows promising results for low-resource NMT, highlighting the importance of domain-aware training and specialized datasets.

What Problem Does This Paper Address?

India is a land of immense linguistic diversity, with hundreds of mother tongues. However, many indigenous and tribal languages, like Bhili, are critically endangered due to a severe lack of digitized linguistic resources, especially parallel corpora. This scarcity hinders the development of effective machine translation (MT) systems, which are crucial for communication, social inclusion, and equitable access to essential services. While advancements in multilingual language models (LLMs) have been significant, they often struggle with underrepresented languages due to biases from high-resource pretraining data. This paper aims to bridge this resource gap for Bhili and explore how existing LLMs perform on such low-resource languages.

Methods

The authors introduce the Bhili-Hindi-English Parallel Corpus (BHEPC), a new dataset of 110,000 meticulously curated sentence pairs. The creation process involved expert human translators, starting with a seed set and iteratively expanding it through model-assisted generation and native speaker post-editing. This hybrid approach aims to be scalable for other endangered languages.

To evaluate MT performance, they employ two main paradigms:

  1. In-Context Learning (ICL): LLMs are tested with a few examples (0, 5, or 10 shots) provided in the prompt, without explicit fine-tuning.
  2. Fine-tuning: Models are further trained on the BHEPC dataset.

They use standard MT evaluation metrics like chrF++ and spBLEU (a sentence-level BLEU variant) and conduct cross-domain generalization experiments by fine-tuning models on different domains (education, administration, news) and testing them on others. They also quantify domain similarity using Jensen-Shannon Divergence (JSD). Human evaluation was also performed to correlate automatic metrics with human judgments.

Datasets

  • Bhili-Hindi-English Parallel Corpus (BHEPC): A newly created dataset of 110,000 sentence pairs, covering education, administration, and news domains. This is the first large-scale, gold-standard parallel corpus for Bhili.
  • Existing Hindi Datasets: Used as a source for Hindi sentences, including the Bharat Parallel Corpus Collection (BPCC), government documents, and NCERT textbooks.
  • Evaluation Datasets: A balanced evaluation dataset was constructed from the corpus, along with domain-specific evaluation sets (NCERT, Govt/PMI, Mass Media) for cross-domain analysis.

Key Insights and Ideas

  • Bhili is Severely Underserved: Despite sharing a script with Hindi, Bhili exhibits significant performance degradation in MT, underscoring that script-level similarity alone is insufficient for effective transfer learning.
  • Proprietary Models Lead, but NLLB Shows Promise: Larger, proprietary models (like GPT-4.5 and Gemini) generally outperform open-source models, especially with more in-context examples. However, the fine-tuned NLLB-200 (600M variant) demonstrates competitive performance, making it a strong contender for low-resource NMT.
  • Domain Matters (A Lot): In-domain fine-tuning consistently yields better results than cross-domain fine-tuning. Domain shift significantly impacts translation quality, highlighting the need for domain-aware adaptation strategies.
  • Character-Level Metrics are Key: For morphologically rich, low-resource languages like Bhili, character-level metrics like chrF++ are more effective than word-level metrics (like spBLEU) in capturing translation quality.
  • Cultural Nuances are a Challenge: Models struggle to capture Bhili-specific vocabulary, idiomatic expressions, and cultural references, often substituting them with Hindi or Gujarati equivalents, or omitting them entirely. This emphasizes the need for datasets that encode cultural specificity.

Key Results

  • The BHEPC corpus contains 108,000 training, 1,000 validation, and 1,000 test sentences.
  • Fine-tuned NLLB-200 (600M) consistently outperformed other models in most translation directions, achieving the highest chrF++ scores.
  • Proprietary models like GPT-4.5 and Gemini 2.5 Flash showed superior performance, particularly with increased in-context examples.
  • Translations from Bhili to high-resource languages (English, Hindi) generally performed better than translations into Bhili.
  • Human evaluation showed that chrF++ scores had stronger alignment with human judgments than spBLEU, especially when Bhili was the target language.

Limitations and Future Work

  • Scarcity of Monolingual Data: This limited the study to supervised fine-tuning, restricting the exploration of unsupervised or semi-supervised approaches.
  • Scalability Concerns: The manual effort required for corpus creation raises questions about scalability to the thousands of other low-resource languages. The authors propose their hybrid workflow as a scalable solution.
  • Future Work: The authors suggest exploring augmentation techniques (back-translation, pivot-based transfer), developing dedicated LLMs for low-resource languages like Bhili, and expanding the BHEPC corpus to include more diverse domains like folklore and oral narratives. Further research is also needed on broader generalizability and cross-domain robustness.

Summary updated

BibTeX
@inproceedings{singh2025leveraging,
  author = {Pooja Singh and Shashwat Bhardwaj and Vaibhav Sharma and Sandeep Kumar},
  title = {{Leveraging the Cross-Domain \& Cross-Linguistic Corpus for Low Resource NMT: A Case Study On Bhili-Hindi-English Parallel Corpus}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure