Quality Scoring of Source Words in Neural Translation Models

Priyesh Jain, Sunita Sarawagi, Tushar TomarEMNLP 2022 · OpenReview

Abstract

Neural Machine Translation (NMT) systems, while fluent, often suffer from hallucinations or omissions that are difficult for non-native speakers to detect. We present Target-lift, a reference-free method for identifying omitted or mistranslated source words. By calculating the "lift" in probability provided by a generated translation relative to a source language model, Target-lift identifies errors with high precision. Experimental results across English-German, Chinese-English, and English-Hindi show that Target-lift outperforms existing state-of-the-art contrastive conditioning methods by up to 27.1 F1 points while being 4 to 8 times faster.

Introduction

Modern NMT systems frequently produce translations that are fluent but unfaithful to the source text. These errors, particularly omissions (dropping content) and hallucinations (adding content), are often invisible to users who do not speak the target language. Existing error detection methods generally rely on supervised models trained on synthetic data—which often fail to generalize—or computationally expensive contrastive conditioning techniques that require multiple model passes per sentence.

This research addresses the need for an efficient, "reference-free" word-level feedback mechanism. Such a tool would allow users to identify specific source words that were likely mishandled, enabling them to reformulate their input for better translation quality.

Methodology: The Target-lift Metric

The core innovation is the Target-lift metric. The underlying intuition is that a source word $x_i$ is correctly translated if the resulting translation $\hat{y}$ significantly increases the probability of $x_i$ appearing in the source sequence.

Target-lift architecture: comparing unconditional and conditional probabilities to identify mistranslated words.

The metric compares two distinct probabilities:

  1. Unconditional Probability: The likelihood of word $x_i$ given only the preceding source context ($x_{<i}$), calculated using a standard Causal Language Model (LM).
  2. Conditional Probability: The likelihood of $x_i$ given both the preceding source context and the generated translation $\hat{y}$, calculated using a reverse NMT model (translating from target back to source).

The "Lift" is defined as the difference between these two values: $$q(x_i) = Pr(x_i | \hat{y}, x_{<i}) - Pr(x_i | x_{<i}$$

By subtracting the unconditional probability, the method filters out "background noise" from frequent words, focusing strictly on the translation's contribution to the word's likelihood. To ensure comparability between different models, temperature scaling is applied for calibration.

Experimental Setup

The method was evaluated on three language pairs:

  • English-German (En-De): WMT 2020 news data with MQM annotations.
  • Chinese-English (Zh-En): WMT 2020 data with MQM annotations.
  • English-Hindi (En-Hi): A new benchmark of 2,000 human-annotated sentences from the IITB parallel corpus.

Target-lift utilizes off-the-shelf, pre-trained models (e.g., mBART50), requiring no additional fine-tuning or synthetic data generation.

Results and Discussion

Target-lift demonstrates significant improvements in both accuracy and efficiency over previous state-of-the-art methods.

Comparison of F1 scores across three language pairs, showing Target-lift's superior performance.

Performance Gains

As shown in Figure 5, Target-lift consistently achieves higher F1 scores across all tested language pairs. The gains are particularly notable in the English-Hindi task, where it outperformed contrastive conditioning by 27.1 points. ROC curve analysis further confirms that Target-lift is a more reliable indicator of quality across various detection thresholds.

Efficiency and Deployment

A key advantage of Target-lift is its speed. Unlike contrastive conditioning, which requires $N$ model passes (where $N$ is the number of words), Target-lift requires only two passes (one for the reverse NMT and one for the LM). This results in a 4x to 9x speedup, reducing inference time from nearly 400ms to 80ms per sentence for English-German, making it suitable for real-time applications.

Conclusion and Future Work

Target-lift provides a robust, zero-shot framework for word-level quality estimation in NMT. While highly effective, the model faces challenges with idiomatic expressions and subtle connectives (like prepositions) that naturally have high unconditional probabilities. Future research will explore more sophisticated masking strategies and contextual dependency modeling to further refine error detection in these complex cases.

Summary updated

BibTeX
@inproceedings{jain2022quality,
  author = {Priyesh Jain and Sunita Sarawagi and Tushar Tomar},
  title = {{Quality Scoring of Source Words in Neural Translation Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=L-nuyWHa7m},
}
Figure