TL;DR
This paper introduces novel methods for estimating the quality of machine translations without needing access to the original translation system. By leveraging attention weights from a neural machine translation (NMT) model and cross-lingual embeddings, they achieve competitive results, particularly for the German-English language pair. Their approach offers a way to assess translation quality using only the translated text and a pre-trained NMT model.
What Problem Does This Paper Address?
Machine translation has made significant strides, but many systems still produce unreliable translations. Accurately assessing the quality of these translations is crucial for downstream applications and for identifying areas for improvement. Traditional quality estimation (QE) methods often rely on features derived from the translation system itself, making them "system-dependent." This paper tackles the challenge of developing QE methods that are "system-agnostic," meaning they can evaluate translations regardless of the system that produced them, and can even work with human translations. They aim to achieve this by extracting features from the internal workings of NMT models, specifically attention mechanisms, and by using cross-lingual embeddings to compare source and target segments.
Methods
The core of this paper lies in two main feature extraction techniques used to train a regression model for quality estimation:
- Force-Decoded Attention Weights: Standard NMT models use attention mechanisms to align input and output tokens during translation. However, to use these attention weights for QE on any translation (not just those produced by the specific NMT model), the authors employ "force-decoding." This involves using an NMT model to predict the probability of each token in a given translation, effectively "forcing" the model to follow the provided translation. This process allows them to extract attention weights that reflect the alignment between the source and the provided translation. They then derive several metrics from these attention weights, including:
- Coverage Deviation Penalty (CDP): Penalizes input tokens with too little or too much attention.
- Absentmindedness Penalties (APin, APout): Measure the dispersion (entropy) of attention distributions for input and output tokens.
- Total: The sum of the above three metrics.
- APratio: A modification calculating the ratio between input and output absentmindedness penalties.
- Cross-lingual Embeddings: This approach aims to measure the similarity between source and target segments by aligning their embedding spaces. They explore three types of embeddings:
- Word-level embeddings: Trained on unigram words.
- Phrase-level embeddings: Concatenated words (up to three) into phrases.
- BPE-level embeddings: Utilize sub-word units from pre-trained NMT models. They learn joint cross-lingual vector spaces using the MUSE (Multilingual Unsupervised or Supervised word Embeddings) method. This allows them to compute an embedding-enhanced BLEU score (BLEU2VEC) where n-gram precisions are weighted by the cosine similarity of their cross-lingual embeddings.
These features are then fed into a Random Forest regression model for quality estimation.
Datasets
The paper uses data from the WMT18 shared task on sentence-level quality estimation. The language pairs and systems involved are:
- German-English (NMT and SMT)
- English-German (NMT and SMT)
- English-Latvian (SMT)
- English-Czech (SMT)
The number of sentences for each language pair and system is detailed in Table 1 of the paper. No new datasets were introduced.
Key Insights and Ideas
- Force-decoding unlocks system-agnostic attention: The ability to extract meaningful attention weights from any translation, not just those produced by the training NMT model, is a significant contribution. This bypasses the need to have direct access to the translation system.
- Attention weights are powerful QE features: The paper demonstrates that features derived from attention weights, especially when combined with other features, yield strong performance, sometimes outperforming baseline methods.
- Cross-lingual embeddings alone are less effective: Initial experiments showed that cross-lingual embeddings as standalone features provided only a near-zero correlation with quality scores. Their utility is primarily realized when combined with other features.
- Domain matters for QE performance: The German-English language pair showed significantly higher correlation coefficients compared to other pairs. The authors hypothesize this is due to the domain of the data (pharmaceutical vs. IT), suggesting domain-specific QE models might be more effective.
- NMT system choice has minimal impact (for general-domain models): When comparing two different NMT systems for German-English, the choice of system had a negligible impact on the QE performance of their models, suggesting that general-domain NMT models are sufficient for extracting useful attention features.
Key Results
The paper reports results using the Pearson correlation coefficient as the primary evaluation metric.
- Strong performance with combined features: Models combining attention weights with either baseline QE features (QuEst) or cross-lingual embeddings (QuEst+AttW+CrEmb3) achieved the highest Pearson correlation coefficients across most language pairs.
- German-English superiority: For the German-English language pair, their proposed models (AttW and AttW+BPE) significantly outperformed the baseline model, showing results more than twice that of the baseline.
- AttW and AttW+BPE as resource-efficient models: These models are highlighted for not requiring additional resources beyond an NMT system (for attention weights and BPE embeddings).
Limitations and Future Work
- Domain-specific analysis: While the authors observed domain-specific performance differences (particularly for German-English), a more thorough investigation into how domain affects QE performance is left for future work.
- NMT system choice: While general-domain NMT systems seem sufficient, the paper suggests further exploration into the impact of NMT system choice, especially for domain-specific systems.
- Further experimentation with attention weights: The authors plan to experiment further with attention weights, specifically comparing internal vs. force-decoded attention weights for in-domain systems.
Summary updated