Abstract
Neural Machine Translation (NMT) systems, while fluent, often suffer from hallucinations or omissions that are difficult for non-native speakers to detect. We present Target-lift, a reference-free method for identifying omitted or mistranslated source words. By calculating the "lift" in probability provided by a generated translation relative to a source language model, Target-lift identifies errors with high precision. Experimental results across English-German, Chinese-English, and English-Hindi show that Target-lift outperforms existing state-of-the-art contrastive conditioning methods by up to 27.1 F1 points while being 4 to 8 times faster.
Introduction
Modern NMT systems frequently produce translations that are fluent but unfaithful to the source text. These errors, particularly omissions (dropping content) and hallucinations (adding content), are often invisible to users who do not speak the target language. Existing error detection methods generally rely on supervised models trained on synthetic data—which often fail to generalize—or computationally expensive contrastive conditioning techniques that require multiple model passes per sentence.
This research addresses the need for an efficient, "reference-free" word-level feedback mechanism. Such a tool would allow users to identify specific source words that were likely mishandled, enabling them to reformulate their input for better translation quality.
Methodology: The Target-lift Metric
The core innovation is the Target-lift metric. The underlying intuition is that a source word $x_i$ is correctly translated if the resulting translation $\hat{y}$ significantly increases the probability of $x_i$ appearing in the source sequence.
The metric compares two distinct probabilities:
- Unconditional Probability: The likelihood of word $x_i$ given only the preceding source context ($x_{<i}$), calculated using a standard Causal Language Model (LM).
- Conditional Probability: The likelihood of $x_i$ given both the preceding source context and the generated translation $\hat{y}$, calculated using a reverse NMT model (translating from target back to source).
The "Lift" is defined as the difference between these two values: $$q(x_i) = Pr(x_i | \hat{y}, x_{<i}) - Pr(x_i | x_{<i}$$
By subtracting the unconditional probability, the method filters out "background noise" from frequent words, focusing strictly on the translation's contribution to the word's likelihood. To ensure comparability between different models, temperature scaling is applied for calibration.
Experimental Setup
The method was evaluated on three language pairs:
- English-German (En-De): WMT 2020 news data with MQM annotations.
- Chinese-English (Zh-En): WMT 2020 data with MQM annotations.
- English-Hindi (En-Hi): A new benchmark of 2,000 human-annotated sentences from the IITB parallel corpus.
Target-lift utilizes off-the-shelf, pre-trained models (e.g., mBART50), requiring no additional fine-tuning or synthetic data generation.
Results and Discussion
Target-lift demonstrates significant improvements in both accuracy and efficiency over previous state-of-the-art methods.
Performance Gains
As shown in Figure 5, Target-lift consistently achieves higher F1 scores across all tested language pairs. The gains are particularly notable in the English-Hindi task, where it outperformed contrastive conditioning by 27.1 points. ROC curve analysis further confirms that Target-lift is a more reliable indicator of quality across various detection thresholds.
Efficiency and Deployment
A key advantage of Target-lift is its speed. Unlike contrastive conditioning, which requires $N$ model passes (where $N$ is the number of words), Target-lift requires only two passes (one for the reverse NMT and one for the LM). This results in a 4x to 9x speedup, reducing inference time from nearly 400ms to 80ms per sentence for English-German, making it suitable for real-time applications.
Conclusion and Future Work
Target-lift provides a robust, zero-shot framework for word-level quality estimation in NMT. While highly effective, the model faces challenges with idiomatic expressions and subtle connectives (like prepositions) that naturally have high unconditional probabilities. Future research will explore more sophisticated masking strategies and contextual dependency modeling to further refine error detection in these complex cases.
Summary updated