Wikidata-Driven Entity-Aware Translation: Boosting LLMs with External Knowledge

Lu XuACL 2025

TL;DR

This paper tackles the challenge of accurately translating named entities (like people, places, and products) in machine translation, a known weakness of Large Language Models (LLMs). They propose and demonstrate an effective approach that leverages Wikidata, a vast knowledge base, to enrich LLMs with external entity information, significantly boosting translation quality, especially for culturally specific or low-resource languages.

What Problem Does This Paper Address?

Machine translation (MT) has seen remarkable progress, largely thanks to the advent of Large Language Models (LLMs). However, a persistent challenge is the accurate translation of named entities – proper nouns referring to people, landmarks, cultural artifacts, etc. LLMs, despite their general translation prowess, often falter with these entities because:

  • Rarity in Training Data: Named entities are often infrequent in standard training corpora, leading to poor internal representations.
  • Ambiguity: Resolving entity references often requires contextual or world knowledge that LLMs may lack.
  • Cultural Specificity: Entities tied to specific cultures or domains demand specialized knowledge beyond general MT capabilities.

This problem is exacerbated when translating between languages with different writing systems or cultural contexts, where phonetic adaptation and cultural localization are crucial. The paper specifically addresses the SemEval-2025 Task 2, which focuses on this entity-aware machine translation problem.

Methods

The core of the proposed solution is to augment LLMs with external knowledge from Wikidata. They explore two main strategies:

  1. Gold Entity Knowledge Integration: This method utilizes "gold" Wikidata IDs provided in the dataset. The system queries the Wikidata API to retrieve entity names in both source and target languages. This retrieved information is then incorporated into the LLM's translation prompt, guiding it to use the correct entity representations.
  2. Automatic Entity Detection and Knowledge Retrieval (Relik-based): Recognizing that gold annotations are impractical for real-world applications, this approach uses Relik, an information extraction tool, to automatically identify potential named entities in the source text. For each detected entity, Wikidata is queried to retrieve its multilingual representations. These automatically retrieved mappings are then used to enhance the translation prompt.

Additionally, they implemented a language-specific post-processing step using the zhconv tool to convert Simplified Chinese characters to Traditional Chinese in the output, addressing a common issue where LLMs produce mixed character sets.

Datasets

The paper evaluates its system on the SemEval-2025 Task 2 dataset. This dataset is designed to highlight the challenges of entity-aware machine translation. The authors do not introduce a new dataset but rather leverage an existing benchmark for their experiments.

Key Insights and Ideas

  • Wikidata is a Game Changer: Integrating external knowledge from Wikidata dramatically improves entity translation accuracy, often by over 50 percentage points, far surpassing baseline LLMs relying solely on parametric knowledge.
  • Target Language Entity Info is Crucial: Simply knowing the source language entity isn't enough. The model needs target language entity information to perform accurate translations.
  • Automatic is Practical: The Relik-based approach, which automatically identifies and links entities without gold annotations, achieves competitive results and is a practical solution for real-world scenarios.
  • Low-Resource Languages Benefit Most: The impact of entity knowledge integration is particularly pronounced for low-resource languages (like Thai), where it can compensate for limited training data and significantly boost performance, sometimes even surpassing richer-resource languages.
  • LLMs Can "Answer" Instead of Translate: When dealing with question-like sentences in the dataset, LLMs sometimes attempted to answer the question rather than translate it, highlighting the need for careful prompt engineering.
  • Prompt Language Matters Less Than Knowledge: Experimenting with prompting in the target language versus English showed minimal and inconsistent effects, suggesting that the availability of external entity knowledge is a far stronger determinant of performance.

Key Results

  • Significant m-ETA Improvement: The "gold" entity integration approach achieved m-ETA scores of up to 89.0%, a substantial increase from baseline LLMs (around 0.257-0.317). The Relik-based approach achieved m-ETA scores of 0.714-0.728.
  • High Overall Ranking: The gold entity system ranked 3rd overall and 1st among non-finetuned systems on the SemEval-2025 Task 2 leaderboard.
  • Practical Solution Outperforms Baselines: The Relik-based system, while not officially submitted, demonstrated competitive results and significantly outperformed non-gold, non-finetuned systems.
  • Traditional Chinese Improvement: The zhconv post-processing step improved m-ETA scores by 6-16 percentage points for Traditional Chinese translations, with GPT-40-mini seeing an increase from 64.61% to 80.64%.

Limitations and Future Work

  • Wikidata Dependency: The system's performance is contingent on the quality and coverage of Wikidata. For low-resource languages or specialized domains, Wikidata might lack comprehensive information, leading to potential inconsistencies.
  • Error Propagation (Relik): The Relik-based approach is susceptible to error propagation. Inaccuracies in entity detection or linking can directly impact translation quality, with analysis showing 15-20% of Relik errors stemming from linking failures.
  • Language-Specific Complexity: Post-processing solutions like Chinese script conversion add complexity and don't generalize well across all language pairs, requiring separate pipelines.
  • Latency: The approach requires multiple API calls to external services, which can introduce latency, potentially problematic for real-time applications.

Future work could explore more robust entity linking mechanisms, investigate methods to mitigate error propagation, and develop more generalized post-processing techniques.

Summary updated

BibTeX
@inproceedings{xu2025wikidata,
  author = {Lu Xu},
  title = {{Wikidata-Driven Entity-Aware Translation: Boosting LLMs with External Knowledge}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure