Abstract
Quality Estimation (QE) remains a critical challenge in Machine Translation (MT), aiming to evaluate translation quality without human-written references. This paper details the joint submission from Johns Hopkins University and Microsoft for the WMT21 QE Shared Task. The authors propose a novel adaptation of the Levenshtein Transformer (LevT), a non-autoregressive model, for word-level QE. By integrating massively multilingual pre-training (M2M-100) with a three-stage finetuning pipeline—incorporating synthetic data generation and human-labeled refinement—the system achieved state-of-the-art results, notably securing the top rank for the English-German language pair.
Introduction
In the field of Machine Translation, Quality Estimation (QE) is often described as the "holy grail" of evaluation. Unlike traditional metrics like BLEU or METEOR, QE attempts to predict the quality of a translation in a vacuum, without access to a gold-standard reference. Word-level QE is particularly demanding, as it requires identifying specific errors—substitutions, deletions, or insertions—at the token level.
Historically, word-level QE has faced significant hurdles, including the difficulty of utilizing bidirectional context effectively, the mismatch between subword-level modeling and word-level human labels, and the inherent complexity of predicting "gaps" where words might be missing.
Methodology: The Levenshtein Transformer
The core innovation of this submission is the repurposing of the Levenshtein Transformer (LevT). While standard Transformers generate text autoregressively from left to right, LevT is a non-autoregressive model designed to iteratively refine text through deletion and insertion operations. This edit-based architecture is naturally suited for QE, as its internal mechanisms for predicting edits map directly to the "OK" and "BAD" labels used in quality estimation tasks.
The Three-Stage Pipeline
The authors implemented a robust three-stage training strategy to maximize performance:
- Stage 1: Translation Pre-training: The model was initialized with weights from M2M-100, a massively pre-trained multilingual model, and further trained on standard parallel translation data. This ensures the model possesses a strong foundational understanding of the target language pairs.
- Stage 2: Synthetic Finetuning: To overcome the scarcity of human-labeled QE data, the team generated "pseudo-post-editing" triplets. They utilized Multi-view Pseudo Post-Editing (MVPPE), which ensembles different translation and paraphrasing views to create realistic error patterns, teaching the model to recognize common MT failures.
- Stage 3: Human Finetuning: The final stage involved finetuning on the limited gold-standard human post-edited data provided by the WMT21 competition. To address the severe label imbalance (where "OK" tags vastly outnumber "BAD" tags), a hyperparameter $\mu$ was introduced to upweight the loss for "BAD" predictions.
Results and Performance
The JHU-Microsoft system demonstrated substantial improvements over the standard OpenKiwi-XLM baseline across all language pairs. As shown in Figure 2, the system achieved a Matthews Correlation Coefficient (MCC) of 0.504 for English-German, a significant leap from the baseline's 0.337. Similar gains were observed in English-Chinese and various low-resource pairs.
Detailed Error Analysis
While the system performed exceptionally well overall, a granular analysis of the results reveals the specific strengths and weaknesses of current word-level QE models. The system is highly proficient at identifying "OK" tags for both words and gaps, often achieving F1 scores above 0.95.
However, identifying "BAD" gaps—locations where a word is missing in the translation—remains a significant challenge. As detailed in Figure 6, the F1 scores for "GAP BAD" predictions range between 0.12 and 0.28, highlighting a persistent difficulty in detecting omissions compared to identifying incorrect words already present in the text.
Conclusion
The JHU-Microsoft submission proves that non-autoregressive architectures like the Levenshtein Transformer are powerful tools for evaluation tasks. By treating QE as an edit-prediction problem and leveraging large-scale multilingual pre-training alongside diverse synthetic data, the authors have set a new benchmark for word-level quality estimation. Future work will likely focus on improving the detection of missing words (the "BAD gap" problem) and further refining multilingual transfer strategies.
Summary updated