Zero at SemEval-2025 Task 2: Entity-Aware Machine Translation: Fine-Tuning NLLB for Improved Named Entity Translation

Revanth Gundam, Abhinav Marri, Advaith Malladi, Radhika MamidiACL 2025

TL;DR

This paper tackles the persistent challenge of accurately translating Named Entities (NEs) in Machine Translation (MT). The authors propose a novel approach by fine-tuning a pre-trained NLLB model using a custom "silver" dataset generated via Google Translate, demonstrating that specialized fine-tuning can significantly improve NE translation quality without relying on massive general-purpose models.

What Problem Does This Paper Address?

Machine Translation (MT) has made incredible strides, but accurately translating Named Entities (NEs) – like people's names, places, and organizations – remains a significant hurdle. This is due to their rarity, ambiguity, and language-specific variations. Traditional MT models often struggle with NEs because they might not have direct equivalents across languages, leading to mistranslations that can cause misinformation or loss of meaning, especially critical in fields like news, medical, and legal translation. Existing approaches often rely on large lexicons or parallel corpora, which can fail to generalize to unseen entities. This paper aims to improve Entity-Aware Machine Translation (EA-MT) by developing a more robust and adaptable translation system for NEs.

Methods

The core of this paper's methodology involves two key steps:

  1. Silver Dataset Creation: Instead of relying on existing datasets that might be limited or generated by general-purpose language models (which are optimized for broad language modeling rather than specific translation tasks), the authors created their own "silver" dataset. They achieved this by using the Google Translate API to translate sentences from the original task dataset. This approach leverages a dedicated translation system to generate higher-quality, machine-translation-specific NE translations.
  2. Fine-Tuning NLLB with LoRA: The authors chose to fine-tune the facebook/nllb-200-distilled-600M model, a powerful pre-trained MT model that supports a wide range of languages. To make this fine-tuning process more efficient, they employed LoRA (Low-Rank Adaptation). LoRA is a parameter-efficient fine-tuning technique that significantly reduces memory requirements compared to full fine-tuning, allowing for specialized models without the prohibitive computational cost. The NLLB model was fine-tuned individually for each target language using the custom silver dataset.

Datasets

  • Training Dataset: A custom "silver" dataset was created by translating sentences from the original task dataset using the Google Translate API. This dataset contains source English sentences, target language translations, and associated Wikidata IDs for entities.
  • Evaluation Dataset: The paper refers to the dataset provided for the SemEval-2025 Task 2, which includes evaluation data with source English sentences, translated target sentences, entity types, and Wikidata IDs.

Key Insights and Ideas

  • Specialization Trumps Generalization: The paper strongly suggests that fine-tuning a specialized MT model on a carefully curated dataset can outperform general-purpose large language models for specific tasks like NE translation. This highlights the value of domain-specific adaptation.
  • LoRA for Efficiency: The successful application of LoRA demonstrates its effectiveness in enabling efficient fine-tuning of large MT models for specific tasks, making advanced customization more accessible.
  • Limitations of BLEU: The authors provide compelling examples showing how the BLEU score can be misleading for NE translation. A high BLEU score can mask significant NE translation errors, underscoring the need for more nuanced evaluation metrics like COMET and M-ETA.
  • Morphological Complexity and Evaluation: For languages like Japanese and Korean, lower BLEU scores were observed, but higher COMET and M-ETA scores indicated better semantic preservation. This suggests that BLEU might not fully capture translation adequacy in morphologically rich languages, and a combination of metrics is crucial.

Key Results

The fine-tuned NLLB model showed promising results across various languages:

  • High Performance Languages: Spanish (59.14 BLEU), Italian (54.52 BLEU), and Turkish (49.38 BLEU) achieved the highest BLEU scores. Spanish and Turkish also performed very well in the harmonic mean of COMET and M-ETA scores (46.46 and 46.50 respectively).
  • Moderate Performance: Arabic (47.24 BLEU) and German (44.98 BLEU) showed competitive results.
  • Challenging Languages: Japanese (1.71 BLEU) and Chinese (Traditional) (0.11 BLEU) struggled significantly with BLEU scores. However, Japanese and Korean showed improved performance in COMET and M-ETA scores (35.28 and 35.97 respectively) compared to their BLEU scores, indicating better semantic preservation despite word-level inaccuracies.
  • Lowest Performance: Chinese (Traditional) (8.41) and Thai (13.75) consistently showed the lowest scores across both BLEU and COMET/M-ETA metrics, highlighting the inherent difficulties in translating these languages.

Limitations and Future Work

  • Implicit Entity Awareness: The current approach fine-tunes the model indirectly on entity awareness. There's no explicit mechanism to ensure entities are treated distinctly from regular words. Future work could explore explicit methods to enforce this.
  • Reliance on Google Translate: The quality of the "silver" dataset is dependent on Google Translate's accuracy. While generally good, it might not always provide perfect NE translations, potentially introducing noise into the training data. Future work could involve human verification or more sophisticated dataset generation techniques.
  • BLEU Score Limitations: The paper explicitly points out the shortcomings of BLEU for evaluating NE translation and advocates for the use of metrics like COMET and M-ETA. This suggests that future research should continue to explore and develop better evaluation methodologies for NE translation.

Summary updated

BibTeX
@inproceedings{gundam2025zero,
  author = {Revanth Gundam and Abhinav Marri and Advaith Malladi and Radhika Mamidi},
  title = {{Zero at SemEval-2025 Task 2: Entity-Aware Machine Translation: Fine-Tuning NLLB for Improved Named Entity Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure