Challenges in Urdu Machine Translation

Abdul Basit, Abdul Hameed Azeemi, Agha Ali RazaACL 2024

TL;DR

This paper investigates the performance of four different types of neural machine translation models (LLM, bilingual, and two multilingual) for English-to-Urdu translation. They found that IndicTrans2, a model specifically designed for Indic languages, significantly outperforms others, highlighting the importance of specialized training data and architectures for low-resource languages.

What Problem Does This Paper Address?

Neural Machine Translation (NMT) has revolutionized translation, but its effectiveness often diminishes for low-resource languages like Urdu. This is primarily due to a lack of sufficient high-quality training data compared to high-resource languages. While large language models (LLMs) like GPT-3.5 show promise, their performance on Urdu is not well-understood, and it's unclear how they stack up against specialized bilingual and multilingual NMT models. This paper aims to bridge this gap by comprehensively evaluating different NMT approaches for English-to-Urdu translation and identifying specific challenges.

Methods

The researchers evaluated four distinct types of machine translation models:

  • Large Language Model (LLM): GPT-3.5, accessed via API, with a specific translation prompt and contextual information.
  • Bilingual Model: opus-mt-en-ur, a model specifically trained for English-to-Urdu translation.
  • Multilingual Models:
    • NLLB: A large-scale multilingual model trained on 200 languages, including both high-resource and low-resource ones.
  • IndicTrans2: A specialized multilingual model designed for 22 Indic languages, including Urdu.

The evaluation involved preprocessing test data to ensure standardized and reliable results. They used SacreBLEU and CHRF++ metrics to assess translation quality. To understand performance differences, they also analyzed the Zipf distribution of Urdu words in the training data of the models.

Datasets

The models were evaluated on five publicly available test datasets:

  • tatoteba-test.eng-urd: From the Tatoeba Translation Challenge.
  • Flores 101: A benchmark for low-resource languages, with the Urdu subset used.
  • Mann Ki Baat: Contains Urdu language content from speeches by the Indian Prime Minister.
  • UMC005: A parallel corpus of English-Urdu alignments from various texts (Quran, Bible, etc.).
  • TED Talk test dataset: Used for more challenging evaluations.

The paper also discusses the training data characteristics of the models, noting that IndicTrans2 uses a combination of auxiliary training (back-translation) and downstream training on augmented parallel corpora, including human-generated seed data.

Key Insights and Ideas

  • Specialization Wins for Low-Resource: IndicTrans2, a model specifically trained for Indic languages, significantly outperformed general-purpose LLMs and other multilingual models for English-to-Urdu translation. This underscores the critical role of domain-specific training data and architectures for low-resource languages.
  • Data Distribution Matters: The superior performance of IndicTrans2 is linked to its Urdu word distribution in the training data, which exhibits a "better long tail" (more diverse vocabulary usage) compared to other datasets. This suggests that optimizing training data composition is crucial.
  • LLMs are Good, but Not Always Best: While GPT-3.5 showed strong performance, it lagged behind the specialized multilingual models, indicating that general-purpose LLMs, despite their scale, may not inherently excel at low-resource translation without specific fine-tuning or data.
  • Challenges Persist: Despite advancements, Urdu NMT still faces issues like Named Entity Recognition (NER) errors, mistranslation, word repetition, literal translations, word omissions, and transliteration errors, even with advanced models.

Key Results

  • SacreBLEU Scores: IndicTrans2 achieved the highest SacreBLEU scores across most datasets, significantly outperforming GPT-3.5, NLLB, and opus-mt-en-ur. For example, on the TED Talk dataset, IndicTrans2 scored 16.50, while GPT-3.5 scored 12.29 and NLLB scored 19.55 (though NLLB performed better on some other datasets).
  • CHRF++ Scores: IndicTrans2 also demonstrated superior CHRF++ scores, indicating better translation quality.
  • Zipf Distribution Analysis: The Zipf distribution analysis revealed that IndicTrans2 and NLLB had more similar distributions to ideal language models, with IndicTrans2 showing a better long tail, contributing to its higher performance.

Limitations and Future Work

Limitations:

  • The study focused on English-to-Urdu translation; other language pairs or directions were not explored.
  • While IndicTrans2 performed best, some challenges like word repetition were observed across all models.
  • The evaluation was limited to specific datasets, and performance might vary on domain-specific data.

Future Work:

  • Extend the evaluation to additional domain-specific datasets.
  • Investigate other low-resource Indic languages to uncover further translation challenges.
  • Focus on developing high-quality Urdu training datasets and refining model training methods.
  • Incorporate more robust evaluation metrics for Urdu NMT.

Summary updated

BibTeX
@inproceedings{basit2024challenges,
  author = {Abdul Basit and Abdul Hameed Azeemi and Agha Ali Raza},
  title = {{Challenges in Urdu Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure