VerbaNexAI at SemEval-2025 Task 2: Enhancing Entity-Aware Translation with Wikidata-Enriched MarianMT

Daniel Peña Gnecco, Juan Carlos Martinez Santos, Edwin PuertasACL 2025

TL;DR

This paper tackles the challenge of Entity-Aware Machine Translation (EA-MT) for English-to-Spanish, aiming to improve the translation of named entities like movie titles and landmarks. While their system achieved good general fluency (COMET score of 87.09), it struggled with entity precision (M-ETA score of 24.62), ranking last among 34 systems, highlighting the limitations of static knowledge enrichment for this task.

What Problem Does This Paper Address?

Translating named entities accurately across languages is a persistent challenge in Natural Language Processing (NLP). Traditional Neural Machine Translation (NMT) models often falter with these entities due to a lack of cultural context and insufficient training data for rare or specific terms. This is particularly true for language pairs like English-to-Spanish, where cultural nuances can significantly alter the meaning or identity of a translated entity. For instance, a literal translation of "The Shawshank Redemption" might not be recognized as a film title by Spanish speakers. SemEval-2025 Task 2 specifically addresses this by focusing on Entity-Aware Machine Translation (EA-MT), aiming to improve the precision of named entity translation.

Methods

The VerbaNexAI system employs a three-stage pipeline:

  1. Data Preprocessing: The EA-MT dataset was cleaned by removing emojis, URLs, mentions, and hashtags. Non-standard characters were eliminated, and accented characters in general text were normalized (e.g., "Á" to "A"), while preserving accents in entity names to maintain their integrity (e.g., "Águila" remained unchanged).
  2. Wikidata Enrichment: To combat data scarcity, the training data was enriched with 240,432 bilingual entity pairs extracted from Wikidata. This covered various categories like musical works, foods, plants, books, fictional entities, landmarks, movies, and TV series.
  3. MarianMT Fine-Tuning: The researchers fine-tuned the MarianMT model, chosen for its efficiency and suitability for English-to-Spanish translation under resource constraints. They used a learning rate of 3x10⁻⁵, a batch size of 4 with gradient accumulation, and trained for four epochs using the AdamW optimizer.

The system explicitly chose a static approach, relying on pre-enriched data from Wikidata, and deliberately discarded dynamic Retrieval-Augmented Generation (RAG) methods due to hardware limitations (NVIDIA RTX 3050 GPU with 4GB VRAM).

Datasets

The primary dataset used was the SemEval-2025 Task 2 EA-MT dataset. Additionally, Wikidata was used as a source for enriching this dataset with bilingual entity pairs.

Key Insights and Ideas

  • Static vs. Dynamic: The paper strongly suggests that static knowledge enrichment (like using Wikidata) is insufficient for robust EA-MT. While it helps with familiar entities, it fails to adapt to emerging or context-specific terms, a capability that dynamic, retrieval-augmented approaches possess.
  • Fluency vs. Precision Trade-off: The system achieved a high COMET score (fluency) but a very low M-ETA score (entity accuracy). This indicates a prioritization of general translation quality over the precise handling of named entities, which is the core goal of EA-MT.
  • Hardware Limitations Matter: The authors explicitly link their choice of a static approach and limited fine-tuning epochs to their constrained hardware. This highlights how computational resources can dictate architectural choices and impact performance, especially in resource-intensive tasks like EA-MT.
  • Preprocessing Pitfalls: Aggressive text normalization, while intended to clean data, inadvertently removed diacritics from general text, potentially impacting contextual translation and even entity meaning if not handled carefully.

Key Results

  • Overall Performance: The VerbaNexAI system achieved a COMET score of 87.09, an M-ETA score of 24.62, and an Overall Score of 38.38.
  • Ranking: This performance placed them last among 34 competing systems.
  • Comparison: Leading systems, often leveraging large language models (LLMs) with dynamic RAG techniques, achieved Overall Scores above 90, with M-ETA scores as high as 90.50.
  • Successes: The system successfully translated some well-documented entities, such as "Eagle of St. John" to "Águila de San Juan."
  • Failures: Frequent failures included mistranslations like "Breaking Bad" to "Rompiendo Malo" and cultural errors like "Star Wars" to "Guerras Estelares" instead of the more common "La Guerra de las Galaxias."

Limitations and Future Work

  • Over-reliance on Wikidata: The static nature of Wikidata prevents adaptation to new or domain-specific terms.
  • Lack of Contextual Adaptation: The system cannot dynamically adjust translations, leading to rigid and often incorrect outputs for ambiguous or context-sensitive entities.
  • Preprocessing-induced Errors: Aggressive normalization of text removed diacritics, impacting entity integrity and meaning.
  • Computational Constraints: Limited hardware restricted fine-tuning depth, potentially hindering the model's ability to fully leverage the enriched dataset.

Future work should focus on integrating retrieval-augmented generation (RAG) and adaptive entity-linking techniques for dynamic contextual information. Improving preprocessing strategies to preserve linguistic integrity and increasing computational resources for deeper fine-tuning are also crucial. Exploring meta-learning and heuristic reasoning could further enhance adaptability.

Summary updated

BibTeX
@inproceedings{gnecco2025verbanexai,
  author = {Daniel Peña Gnecco and Juan Carlos Martinez Santos and Edwin Puertas},
  title = {{VerbaNexAI at SemEval-2025 Task 2: Enhancing Entity-Aware Translation with Wikidata-Enriched MarianMT}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure