Estimating Machine Translation Difficulty

Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, Tom KocmiEMNLP 2025

TL;DR

Machine translation systems are getting incredibly good, making it hard to tell them apart or find areas for improvement. This paper introduces a formal way to estimate how difficult a text is to translate, developing new metrics and models that can identify challenging texts. This work is crucial for creating better benchmarks that can truly push the boundaries of machine translation research.

What Problem Does This Paper Address?

As machine translation (MT) systems achieve near-human performance on many benchmarks, it's becoming increasingly difficult to differentiate between top models and identify specific weaknesses. Existing evaluation methods often struggle to distinguish between models when the test sets are too easy. This paper addresses the need for a more nuanced understanding of translation difficulty, aiming to automatically identify texts that pose a challenge to current MT systems. This is important for developing more discriminative evaluation metrics, guiding future research, and constructing more effective training datasets.

Methods

The paper formalizes the task of Translation Difficulty Estimation (TDE), defining a text's difficulty based on the expected quality of its translations. They introduce a novel evaluation metric called Difficulty Estimation Correlation (DEC), which measures how well an estimator can rank texts by difficulty.

They explore several categories of TDE methods:

  • Heuristic-based estimators: Rely on simple text features like text length, word rarity, and syntactic complexity.
  • Learned estimators: Utilize neural models trained to predict translation difficulty. The paper highlights sentinel-src-24 and sentinel-src-25 as their top-performing learned models, which are trained to predict translation quality solely from the source text.
  • LLM-as-a-Judge: Leverages large language models (GPT-4, CommandA) to assess difficulty by prompting them to estimate the proficiency level required for translation.
  • Crowd-based estimators:
    • Artificial Crowd: Translates source texts with multiple MT models and then uses reference-less MT metrics (XCOMET, MetricX) to estimate the quality of these translations.
  • True Crowd: Uses actual ground-truth translations and reference-less MT metrics to establish an upper bound for Artificial Crowd estimators.

Datasets

The primary dataset used for evaluation is the WMT 2024 General MT and Metrics shared tasks test sets. These sets contain source texts translated into multiple languages by both automatic models and human annotators, with quality annotations (ESA and MQM protocols). The authors also mention using data from previous WMT editions (WMT 21, 22, 23) for training their improved models.

Key Insights and Ideas

  • Difficulty is relative: Translation difficulty is not an absolute property of a text but depends on the target language and the specific translator (human or machine).
  • Learned models excel: Models explicitly trained for TDE, particularly sentinel-src-24, significantly outperform traditional heuristic methods and LLM-as-a-Judge approaches.
  • LLMs struggle with difficulty: Despite their general capabilities, LLM-as-a-Judge methods perform poorly in estimating translation difficulty, often being surpassed even by simple heuristics like text length.
  • Artificial Crowd is promising but costly: Artificial Crowd methods show performance comparable to learned models but are considerably more resource-intensive.
  • WMT datasets are too easy: The current WMT test sets are too easy for state-of-the-art MT models, with top systems achieving near-perfect scores across many language pairs. This underscores the need for more challenging benchmarks.
  • Bias in difficulty selection: Difficulty estimators can exhibit bias towards certain MT models, a crucial consideration when creating new benchmarks.

Key Results

  • sentinel-src-24 achieved state-of-the-art performance in difficulty estimation, outperforming all other evaluated methods.
  • When creating difficult benchmarks by selecting the hardest 25% of texts, sentinel-src-24 and Artificial Crowd methods were most effective at identifying texts that challenge current MT systems, leading to lower average human scores and fewer perfect translations.
  • Heuristic-based estimators performed poorly, with Text Length being the only exception that showed some utility.
  • LLM-as-a-Judge methods showed surprisingly low performance, often worse than simple heuristics.
  • True Crowd methods, using ground-truth translations, provided the highest correlation, demonstrating the effectiveness of XCOMET over MetricX.
  • Human translators perceive translation difficulty differently from automatic models, with lower correlations observed between human and MT model scores compared to MT model-to-model correlations.

Limitations and Future Work

  • Concept of difficulty: The paper acknowledges that their definition of difficulty is based on the quality of translations, which is a proxy for the intrinsic difficulty of the source text. Future work could explore the abstract, model-independent notion of translation difficulty more deeply.
  • Impact of target language: While the paper notes that target language can influence difficulty and that LLM-as-a-Judge benefits from this information, a thorough investigation into this aspect is not provided. Future research is encouraged to explore this more thoroughly.
  • WMT test set limitations: The analysis of difficulty estimators was conducted on the WMT 2024 test set, which consists of human-vetted sources. This might lead to an artificial distribution of phenomena. The authors suggest using larger, naturally collected datasets for future analysis.
  • Bias in benchmark creation: The paper identifies a potential bias in difficulty-based benchmark creation, where certain MT models might be disproportionately challenged. Further investigation into this discrepancy and its causes is needed.
  • Model-specific difficulty: The study suggests that difficulty estimators might be more or less aligned with specific MT models. Future work could delve deeper into understanding these model-specific discrepancies.

Summary updated

BibTeX
@inproceedings{proietti2025estimating,
  author = {Lorenzo Proietti and Stefano Perrella and Vilém Zouhar and Roberto Navigli and Tom Kocmi},
  title = {{Estimating Machine Translation Difficulty}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure