DMDTEval: An Evaluation and Analysis of LLMs on Disambiguation in Multi-domain Translation

Zhibo Man, Yuanmeng Chen, Yujie Zhang, Jinan XuEMNLP 2025

TL;DR

Large Language Models (LLMs) struggle with multi-domain translation (MDT) due to word ambiguity across different contexts. This paper introduces DMDTEval, a novel framework for evaluating LLMs' disambiguation abilities in MDT. They propose a new dataset, diverse prompting strategies, and specific metrics to analyze how LLMs handle domain-specific word meanings, offering insights into improving MDT performance.

What Problem Does This Paper Address?

Machine translation (MT) has seen remarkable progress thanks to LLMs. However, when translating across different domains (e.g., legal texts to scientific articles), LLMs often falter. This is primarily because the meaning of words can shift significantly depending on the domain, leading to ambiguity. For instance, the word "system" could mean "framework" (体系) in a legal context or a technical "system" (系统) in a scientific one. Existing LLMs, trained on vast but often general datasets, lack the nuanced understanding to consistently resolve these domain-specific ambiguities, resulting in suboptimal translations. This paper aims to systematically evaluate and understand how well LLMs can disambiguate words in multi-domain translation settings.

Methods

The authors developed a comprehensive evaluation framework called DMDTEval, which comprises three key components:

  • Ambiguous Word Dataset Construction: They created a specialized test set by first building bilingual lexicons for various domains using word alignment tools. Then, they identified words that had different translations across domains, creating a cross-domain ambiguous vocabulary. Finally, they manually annotated sentences in test sets with these ambiguous words.
  • Disambiguation Prompt Strategies: They explored various prompting techniques to guide LLMs. These included:
  • Base Prompt Strategies: Zero-shot (direct translation), Chain-of-Thought (step-by-step reasoning), Few-shot (providing examples), and Reflection (reviewing and regenerating).
  • Disambiguation Prompt Strategies: These built upon the base strategies by incorporating domain information, such as domain tags (e.g., "Translate this sentence in the Law domain") or asking the LLM to infer the domain.
  • Disambiguation Metrics: They introduced "Disambiguation Accuracy," a metric that specifically measures how well an LLM translates ambiguous words according to their domain-specific meanings. This is calculated as the ratio of correctly translated ambiguous words to the total number of ambiguous words. They also used standard metrics like BLEU and COMET for overall translation quality.

Datasets

The study utilized existing multi-domain translation test sets:

  • UM-Corpus (English-to-Chinese): Covering five domains: Education, Law, News, Science, and Spoken.
  • OPUS (German-to-English): Covering five domains: IT, Koran, Laws, Medical, and Subtitles.

For other language pairs (Japanese-English and Korean-English), they used data from the FLORES10 dataset. The paper also mentions the construction of a new, specialized ambiguous word dataset for MDT as part of their contribution.

Key Insights and Ideas

  • Domain Information is Crucial: Incorporating domain information into prompts generally leads to improved translation performance, highlighting its importance for LLMs in MDT.
  • Prompting Matters: Different prompting strategies have varying degrees of effectiveness. Strategies that encourage reasoning (like CoT) or reflection, especially when combined with domain information, show promising results.
  • Domain Sensitivity: Some domains, particularly those with specialized terminology and distinct styles (e.g., Law, Science), are more sensitive to prompt strategies, leading to greater improvements when domain information is provided.
  • Metric Limitations: The study found instances where BLEU scores improved while COMET scores decreased, suggesting that standard metrics might not always fully capture the nuances of disambiguation performance in MDT.
  • Reflection Strategy's Strength: The Reflection strategy, even without explicit domain tags, showed consistent improvements across domains, suggesting its inherent ability to leverage context for disambiguation.

Key Results

The paper reports extensive experiments across four language pairs and 13 domains. Key findings include:

  • Qwen-2.5-7B Performance: Among the tested open-source LLMs, Qwen-2.5-7B demonstrated the best performance on English-Chinese MDT.
  • Prompting Improvements: Disambiguation prompting strategies (T5-T10) generally outperformed their base counterparts (T1-T4). For instance, T7 (CoT + Domain Information) achieved the highest average BLEU and COMET scores in English-Chinese translation.
  • Cross-Lingual Consistency: The observed trends and improvements were consistent across different language directions (English-Chinese, German-English, Japanese-English, Korean-English), validating the robustness of their approach.
  • Fine-tuning Benefits: Fine-tuning LLMs with domain information, even with limited data, further boosted performance, with T5 showing the greatest improvement.

Limitations and Future Work

  • Data Scarcity: The paper acknowledges the ongoing challenge of scarce and potentially noisy multi-domain parallel corpora, which can affect evaluation outcomes.
  • Uneven Data Distribution: The focus on English-Chinese and German-English due to linguistic expertise means other language pairs were primarily used for evaluation, lacking the same depth of manual verification.
  • Future Work: The authors plan to develop improved methods for disambiguation by building upon their proposed dataset, aiming to enhance the robustness and generalization of LLMs in MDT and promote more reliable, context-aware applications. They also aim to address the limitations of data scarcity and uneven distribution by focusing on high-quality data collection and annotation.

Summary updated

BibTeX
@inproceedings{man2025dmdteval,
  author = {Zhibo Man and Yuanmeng Chen and Yujie Zhang and Jinan Xu},
  title = {{DMDTEval: An Evaluation and Analysis of LLMs on Disambiguation in Multi-domain Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure