TL;DR
This paper introduces four novel methods for Entity-Aware Machine Translation (EA-MT), aiming to improve how named entities are translated in machine translation systems. They explore both traditional MT with BERT-based NER and advanced LLM-based approaches, demonstrating that LLMs, especially when augmented with external knowledge and reasoning capabilities, offer superior translation accuracy and contextual coherence.
What Problem Does This Paper Address?
Machine translation (MT) systems often struggle with accurately translating named entities (NEs) – proper nouns like names of people, organizations, and places. The meaning and appropriate translation of NEs can vary significantly across languages and contexts, and errors in NE translation can lead to mistranslations that distort the original meaning. This paper tackles the challenge of Entity-Aware Machine Translation (EA-MT), focusing on enhancing the translation of sentences containing these critical named entities.
Methods
The YNU-HPCC team proposed four distinct methods for EA-MT, broadly categorized into traditional MT and LLM-based approaches:
- Method 1: M2M-100 with BERT: This approach combines the m2m-100 pre-trained translation model with a BERT-based Named Entity Recognition (NER) module. It uses a local cache and online retrieval from Wikidata to find translations for identified NEs, cleaning them with regular expressions and then replacing them in the sentence before translation.
- Method 2: Qwen2.5-32B with M2M-100: To address the computational cost of BERT, this method replaces the NER module with the more efficient Qwen2.5-32B model. It uses carefully crafted prompts and examples to guide the LLM in identifying NEs, which are then fed into the m2m-100 for translation.
- Method 3: Qwen Ner and Translator: This method fully embraces LLMs by using the Qwen model for both NER and translation, aiming to leverage its enhanced contextual understanding and translation capabilities.
- Method 4: Qwen with Reason and Act (ReAct): This innovative approach combines LLMs with reasoning and execution abilities using the LangChain framework. It allows the model to identify NEs, search for translations, and even re-evaluate or search for alternative translations if the initial results are unsatisfactory, enhancing interpretability and reliability.
Datasets
The study utilizes the dataset provided for SemEval-2025 Task 2, which includes training, validation, and prediction data in JSON format. The training and validation datasets cover six languages: Arabic (ar), German (de), Spanish (es), French (fr), Italian (it), and Japanese (ja). Four additional languages – Korean (ko), Thai (th), Turkish (tr), and Traditional Chinese (zh-TW) – are included in the validation data. Each data entry contains fields like source language, target language, source sentence, target sentence, NE source, and Wikidata ID, with some entries also specifying NE types.
Key Insights and Ideas
- LLMs Excel in Contextual Translation: The experiments strongly suggest that LLM-based approaches (Methods 2, 3, and 4) significantly outperform traditional MT methods in terms of translation accuracy and contextual coherence for sentences with named entities.
- External Knowledge is Crucial: Integrating external knowledge sources like Wikidata, either through retrieval or by leveraging LLMs' inherent knowledge, is vital for accurate NE translation.
- Reasoning Enhances Reliability: The ReAct method (Method 4) demonstrates that equipping LLMs with reasoning and action capabilities can lead to more robust and interpretable NE translation, especially when dealing with ambiguity or missing information.
- Trade-offs Exist: While LLMs offer higher accuracy, traditional MT models (Method 1) still hold advantages in terms of speed and deployment cost.
Key Results
The paper presents results using M-ETA (Manual Entity Translation Accuracy) and COMET (Cross-lingual Optimized Metric for Evaluation of Translation) scores.
- Overall Performance: Method 3, which uses the Qwen model for both NER and translation, achieved the best overall M-ETA and COMET scores among the four proposed methods, averaging 94.506 for M-ETA and 89.976 for COMET across all languages. Method 4 also performed strongly, with an average M-ETA of 93.802.
- Language-Specific Variations: Table 2 shows detailed results per language, highlighting that while LLM-based methods generally perform better, there can be language-specific nuances and challenges.
- Human Interpretability: Method 4, despite slightly lower scores than Method 3, significantly improved human interpretability and the credibility of the translation outcomes.
Limitations and Future Work
- Traditional MT Limitations: Traditional MT models, even when augmented with NER, may struggle with contextual nuances and can sometimes alter or re-translate named entities in uncontrolled ways. Fine-tuning these models can lead to a decrease in overall translation ability and poor generalization to unseen entities.
- LLM Fine-tuning: The authors suggest that fine-tuning LLMs on specific languages could further improve performance, especially for languages where the current models underperform.
- Tool Functions in LangChain: Future work will focus on enhancing the tool functions within LangChain to assist LLMs more effectively in translation tasks.
- Combining Techniques: The paper anticipates combining fine-tuning and Retrieval Augmented Generation (RAG) techniques to further boost EA-MT performance.
Summary updated