TL;DR
This paper presents two novel neural network architectures for sentence-level Quality Estimation (QE) of machine translation: one leveraging bidirectional translation knowledge and another incorporating monolingual knowledge via BERT. Their ensemble approach achieved competitive results in the WMT19 QE shared task, ranking 3rd out of 10 submissions.
What Problem Does This Paper Address?
Sentence-level Quality Estimation (QE) aims to predict the quality of machine translation (MT) outputs without needing human references. This is crucial for various applications, such as deciding if a translation is good enough for publication, informing users about translation reliability, or filtering sentences for post-editing. Traditional QE methods relied on handcrafted features and machine learning models. While deep learning has shown promise, this paper addresses the challenge of effectively utilizing translation knowledge and leveraging readily available monolingual data to improve QE performance, especially in scenarios where parallel corpora are scarce.
Methods
The authors propose two main approaches, both built around a two-stage neural architecture: a feature extractor and a quality estimator.
- Bi-directional QE Model: This model uses a modified Transformer architecture (similar to the one used in NMT) as a feature extractor. It employs two symmetric "word predictors": one translating source to target and another translating target to source. These predictors generate "Quality Estimation Feature Vectors" (QEFVs) for each word. The key innovation here is the use of bidirectional translation knowledge, where the source-to-target predictor helps the target-to-source predictor and vice-versa, effectively acting as a form of data augmentation. A Bi-directional Long Short-Term Memory (BiLSTM) network then takes these QEFVs to predict the final quality score.
- BERT-based QE Model: This approach leverages the power of pre-trained language models. It uses a Multilingual-BERT model as the feature extractor, which generates QEFVs from the source and target sentences. To further enhance this, they employ a self-attention Transformer to translate the source sentence into a "pseudo-reference" in the target language. This pseudo-reference and the original target sentence are then fed into the BERT feature extractor. Similar to the first model, a BiLSTM is used as the quality estimator. This method is particularly useful when parallel corpora are limited, as it can effectively utilize monolingual data.
Both models are trained using a joint learning approach, minimizing the Mean Average Error (MAE) between predicted and true quality scores. For pre-training the word predictors in the Bi-directional model, they utilize large-scale parallel corpora in both directions.
Datasets
The experiments were conducted on the English-to-German (en-de) language pair using datasets from the WMT17 and WMT19 sentence-level QE shared tasks.
- WMT17 en-de dataset: Consists of 23,000 training sentences, 1,000 development sentences, and 2,000 test sentences.
- WMT19 en-de dataset: Consists of 13,442 training sentences, 1,000 development sentences, and 1,023 test sentences.
The paper notes that gold standard labels for WMT18 were unobtainable, hence the focus on WMT17 and WMT19.
Key Insights and Ideas
- Leveraging Bidirectional Translation Knowledge: The idea that having two translation models (source-to-target and target-to-source) supervise each other can improve feature extraction is a powerful concept. It suggests that the process of generating translations in both directions can implicitly capture richer linguistic nuances relevant to translation quality.
- Monolingual Data is Powerful: The success of the BERT-based model highlights the significant value of monolingual data. By using pre-trained models like BERT and generating pseudo-references, they effectively tap into vast amounts of readily available text to improve QE performance, especially in low-resource scenarios.
- Ensembling for Robustness: Combining the strengths of different models through ensembling proved to be a winning strategy, leading to top performance on the WMT19 task. This underscores the importance of diverse approaches in tackling complex NLP problems.
Key Results
- On the WMT17 sentence-level QE task, their proposed single models, Bi-directional QE and BERT-based QE (+NMT), outperformed other single models on the primary metric (Pearson correlation).
- Their ensemble model achieved the best results on the WMT17 benchmark dataset.
- For the WMT19 QE task, their ensemble model ranked 3rd out of 10 submissions, demonstrating strong performance in a competitive setting. The Bi-directional QE model alone ranked 4th.
Limitations and Future Work
- Limitations: The paper mentions that the training set for QE is often insufficient for training complex neural models from scratch, necessitating pre-training strategies. The performance of the BERT-based QE model with pseudo-references relies on the quality of the pseudo-reference generation.
- Future Work: The authors express interest in extending their approaches to finer-grained QE tasks, such as phrase-level and word-level quality estimation.
Summary updated