TL;DR
This paper evaluates two competing paradigms for Entity-Aware Machine Translation (EA-MT) as part of the SemEval-2025 Task 2 challenge: fine-tuning a smaller open-source model (NLLB-200) versus prompt engineering a commercial Large Language Model (Gemini 1.5 Flash). The results demonstrate that while overall translation fluency is comparable, Gemini dramatically outperforms the fine-tuned model at accurately translating named entities, even in a zero-shot setup.
What Problem Does This Paper Address?
While modern machine translation (MT) systems produce highly fluent text, they frequently stumble when encountering named entities—such as specific people, locations, and organizations. Entities often require specialized handling, such as transliteration (matching phonetic sounds) or precise domain-specific translation, which standard MT models often miss.
To address this, the field of Entity-Aware Machine Translation (EA-MT) focuses on preserving entity integrity during translation. The authors of this paper investigate a core engineering trade-off: Is it better to bilingually fine-tune a dedicated, open-source translation model augmented with external knowledge graph data, or to leverage the massive, pre-existing internal knowledge of a commercial LLM via clever prompting?
Methods
The team explored two primary methodologies across 10 target languages (with English as the source):
- Supervised Fine-Tuning (SFT): The authors selected the distilled 600M-parameter variant of Meta's NLLB-200 model. To make it entity-aware, they used the
spaCyNamed Entity Recognition (NER) framework to extract entities from the training data, looked up their official translations on Wikidata, and appended these entity pairs as additional training data. - Prompt Engineering: They utilized Google's Gemini 1.5 Flash model via LangChain under two conditions:
- Zero-shot: A minimalist template instructing the model to "preserve entity integrity" with no examples.
- Few-shot: The same template but augmented with 10 curated demonstration pairs from the training data, explicitly mapping entities to their Wikidata IDs.
The systems were evaluated using COMET (for overall sentence-level translation quality) and M-ETA (Manual Entity Translation Accuracy), which are combined via a harmonic mean to calculate an Overall Score.
Datasets
The experiments relied on two main data sources:
- SemEval-2025 Task 2 Dataset: Contains sentence pairs for English translated into 10 languages: Arabic, German, Spanish, French, Italian, Japanese, Korean, Thai, Turkish, and Chinese. Crucially, training data was only provided for the first six languages; the remaining four were evaluation-only. Dataset statistics can be seen below:
- Wikidata: Used to extract 4,587 unique entities and their corresponding translations to augment the NLLB training process. The distribution of these extracted entity types (such as
PERSON,ORG, andGPE) is detailed here:
Key Insights and Ideas
- LLMs are Natural Entity Experts: Gemini's entity translation accuracy (M-ETA) was nearly double that of the fine-tuned NLLB model across almost all languages. This suggests that the massive web-scale pre-training of LLMs creates a superior internal knowledge base for entities that is hard to replicate via small-scale fine-tuning.
- Diminishing Returns for Few-Shot Prompting: Surprisingly, there was very little performance difference between Gemini's zero-shot and few-shot configurations. Simply telling the LLM to "preserve entity integrity" was enough; providing explicit examples yielded negligible gains.
- The European-Asian Language Gap: Both approaches performed significantly better on European languages (Spanish, Italian, French, German) than on Asian languages (Chinese, Japanese, Korean, Thai). English-to-Spanish was the top performer, while English-to-Chinese was an extreme outlier, scoring a dismal ~14.83 overall.
- The Turkish Surprise: Despite having absolutely no training data available for Turkish, Gemini achieved an overall score of over 56—second only to Spanish.
Key Results
As shown in the comprehensive results table below, Gemini dominated the evaluation:
- Fluency vs. Accuracy: Interestingly, sentence-level fluency (COMET) was remarkably tight across all models, hovering in the high 80s and low 90s. The real differentiator was entity accuracy (M-ETA).
- Top Performers: For English-to-Spanish, Gemini 10-shot achieved the highest overall score of 63.45 (with an M-ETA of 48.29), compared to NLLB's overall score of 48.20 (M-ETA of 32.75).
- Italian Disparity: In Italian, Gemini notched an overall score of 54.92, whereas NLLB managed only 22.28, highlighting the massive gap in entity handling.
Limitations and Future Work
The authors noted two main limitations in their study:
- Data Asymmetry: The lack of training data for Chinese, Korean, Thai, and Turkish made it impossible to build baseline fine-tuned NLLB models for those languages.
- Compute Constraints: Due to limited hardware resources, the team could only fine-tune the smallest (distilled 600M) variant of NLLB-200. They note that scaling up to a larger open-source parameter variant might close the performance gap with commercial LLMs.
Summary updated