HausaNLP at SemEval-2025 Task 2: Entity-Aware Fine-tuning vs. Prompt Engineering in Entity-Aware Machine Translation

Abdulhamid Abubakar, Hamidatu Abdulkadir, Ibrahim Rabiu Abdullahi, Abubakar Auwal Khalid, Ahmad Mustapha Wali, Amina Aminu Umar, Maryam Bala, Sani Abdullahi Sani, Ibrahim Said Ahmad, Shamsuddeen Hassan Muhammad, Idris Abdulmumin, Vukosi Marivate

TL;DR

This paper evaluates two competing paradigms for Entity-Aware Machine Translation (EA-MT) as part of the SemEval-2025 Task 2 challenge: fine-tuning a smaller open-source model (NLLB-200) versus prompt engineering a commercial Large Language Model (Gemini 1.5 Flash). The results demonstrate that while overall translation fluency is comparable, Gemini dramatically outperforms the fine-tuned model at accurately translating named entities, even in a zero-shot setup.

What Problem Does This Paper Address?

While modern machine translation (MT) systems produce highly fluent text, they frequently stumble when encountering named entities—such as specific people, locations, and organizations. Entities often require specialized handling, such as transliteration (matching phonetic sounds) or precise domain-specific translation, which standard MT models often miss.

To address this, the field of Entity-Aware Machine Translation (EA-MT) focuses on preserving entity integrity during translation. The authors of this paper investigate a core engineering trade-off: Is it better to bilingually fine-tune a dedicated, open-source translation model augmented with external knowledge graph data, or to leverage the massive, pre-existing internal knowledge of a commercial LLM via clever prompting?

Methods

The team explored two primary methodologies across 10 target languages (with English as the source):

  1. Supervised Fine-Tuning (SFT): The authors selected the distilled 600M-parameter variant of Meta's NLLB-200 model. To make it entity-aware, they used the spaCy Named Entity Recognition (NER) framework to extract entities from the training data, looked up their official translations on Wikidata, and appended these entity pairs as additional training data.
  2. Prompt Engineering: They utilized Google's Gemini 1.5 Flash model via LangChain under two conditions:
  3. Zero-shot: A minimalist template instructing the model to "preserve entity integrity" with no examples.
  4. Few-shot: The same template but augmented with 10 curated demonstration pairs from the training data, explicitly mapping entities to their Wikidata IDs.

The systems were evaluated using COMET (for overall sentence-level translation quality) and M-ETA (Manual Entity Translation Accuracy), which are combined via a harmonic mean to calculate an Overall Score.

Datasets

The experiments relied on two main data sources:

  • SemEval-2025 Task 2 Dataset: Contains sentence pairs for English translated into 10 languages: Arabic, German, Spanish, French, Italian, Japanese, Korean, Thai, Turkish, and Chinese. Crucially, training data was only provided for the first six languages; the remaining four were evaluation-only. Dataset statistics can be seen below:
Shared task data statistics
  • Wikidata: Used to extract 4,587 unique entities and their corresponding translations to augment the NLLB training process. The distribution of these extracted entity types (such as PERSON, ORG, and GPE) is detailed here:
Entities extracted from training data

Key Insights and Ideas

  • LLMs are Natural Entity Experts: Gemini's entity translation accuracy (M-ETA) was nearly double that of the fine-tuned NLLB model across almost all languages. This suggests that the massive web-scale pre-training of LLMs creates a superior internal knowledge base for entities that is hard to replicate via small-scale fine-tuning.
  • Diminishing Returns for Few-Shot Prompting: Surprisingly, there was very little performance difference between Gemini's zero-shot and few-shot configurations. Simply telling the LLM to "preserve entity integrity" was enough; providing explicit examples yielded negligible gains.
  • The European-Asian Language Gap: Both approaches performed significantly better on European languages (Spanish, Italian, French, German) than on Asian languages (Chinese, Japanese, Korean, Thai). English-to-Spanish was the top performer, while English-to-Chinese was an extreme outlier, scoring a dismal ~14.83 overall.
  • The Turkish Surprise: Despite having absolutely no training data available for Turkish, Gemini achieved an overall score of over 56—second only to Spanish.

Key Results

As shown in the comprehensive results table below, Gemini dominated the evaluation:

M-ETA, COMET, and Overall scores
  • Fluency vs. Accuracy: Interestingly, sentence-level fluency (COMET) was remarkably tight across all models, hovering in the high 80s and low 90s. The real differentiator was entity accuracy (M-ETA).
  • Top Performers: For English-to-Spanish, Gemini 10-shot achieved the highest overall score of 63.45 (with an M-ETA of 48.29), compared to NLLB's overall score of 48.20 (M-ETA of 32.75).
  • Italian Disparity: In Italian, Gemini notched an overall score of 54.92, whereas NLLB managed only 22.28, highlighting the massive gap in entity handling.

Limitations and Future Work

The authors noted two main limitations in their study:

  1. Data Asymmetry: The lack of training data for Chinese, Korean, Thai, and Turkish made it impossible to build baseline fine-tuned NLLB models for those languages.
  2. Compute Constraints: Due to limited hardware resources, the team could only fine-tune the smallest (distilled 600M) variant of NLLB-200. They note that scaling up to a larger open-source parameter variant might close the performance gap with commercial LLMs.

Summary updated

Abstract

This paper presents our findings for SemEval 2025 Task 2, a shared task on entity-aware machine translation (EA-MT). The goal of this task is to develop translation models that can accurately translate English sentences into target languages, with a particular focus on handling named entities, which often pose challenges for MT systems. The task covers 10 target languages with English as the source. In this paper, we describe the different systems we employed, detail our results, and discuss insights gained from our experiments.
BibTeX
@inproceedings{abubakar2025hausanlp,
  author = {Abdulhamid Abubakar and Hamidatu Abdulkadir and Ibrahim Rabiu Abdullahi and Abubakar Auwal Khalid and Ahmad Mustapha Wali and Amina Aminu Umar and Maryam Bala and Sani Abdullahi Sani and Ibrahim Said Ahmad and Shamsuddeen Hassan Muhammad and Idris Abdulmumin and Vukosi Marivate},
  title = {{HausaNLP at SemEval-2025 Task 2: Entity-Aware Fine-tuning vs. Prompt Engineering in Entity-Aware Machine Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure