TL;DR
The WMT 2018 Shared Task on Quality Estimation (QE) explored automatic methods for predicting machine translation (MT) quality without reference translations, across word, phrase, sentence, and document levels. This year's task introduced new challenges like predicting missing words and source word errors, and evaluated systems on both SMT and NMT outputs, revealing insights into QE model robustness and the evolving landscape of MT errors.
What Problem Does This Paper Address?
Predicting the quality of machine translation (MT) output is crucial for downstream applications and for informing users about potential errors. Traditionally, this has relied on comparing MT output to human-generated reference translations. However, this is often infeasible in real-world scenarios. The WMT Shared Task on Quality Estimation aims to develop automatic methods that can predict MT quality without access to reference translations. This year's task specifically focused on:
- Granularity: Evaluating quality at word, phrase, sentence, and document levels.
- MT Architectures: Assessing QE performance on outputs from both Statistical Machine Translation (SMT) and Neural Machine Translation (NMT) systems.
- New Error Types: Investigating the prediction of missing words and identifying source words that lead to target errors.
- Detailed Post-Editing Data: Exploring the utility of fine-grained post-editing information (like keystrokes and time) for QE.
Methods
The paper details the baseline systems and the approaches submitted by participating teams.
- Baseline Systems:
- Sentence-level: Utilized QUEST++ (Specia et al., 2015) with 17 MT system-independent features, trained with a Support Vector Regression (SVR) model.
- Word-level: Employed the MARMOT tool (Logacheva et al., 2016) to extract 28 features, trained using a Linear-Chain Conditional Random Fields (CRF) algorithm.
- Phrase-level: Leveraged MARMOT for features and a CRFSuite tool with a passive-aggressive optimization algorithm.
- Participant Systems: Teams submitted a variety of systems, often building upon previous QE research. Many systems incorporated neural network architectures, including transformers and recurrent neural networks (RNNs), to capture contextual information. Some systems explored unsupervised or minimalist approaches, while others relied on extensive feature engineering.
Datasets
The shared task utilized several datasets and language pairs:
- English-German (EN-DE): IT domain, SMT and NMT outputs.
- German-English (DE-EN): Life sciences domain, SMT and NMT outputs.
- English-Latvian (EN-LV): Life sciences domain, SMT and NMT outputs.
- English-Czech (EN-CS): IT domain, SMT outputs.
- English-French (EN-FR): Sports and outdoor activities domain (product titles/descriptions), NMT outputs (for Task 4).
New benchmarks were introduced for:
- Missing words: Data annotated for deletion errors at training time.
- Source word errors: Data annotated for source words leading to MT errors at the word level.
- Document-level QE: A new corpus with fine-grained error taxonomy and severity judgments.
Key Insights and Ideas
- NMT vs. SMT Outputs: QE systems generally performed better on SMT outputs than NMT outputs for English-German, potentially due to differences in error types or dataset sizes. However, for English-Latvian, the trend was inverted, with QE systems performing better on NMT data.
- Robustness of QE: Despite differences in MT architectures and error distributions, the ranking of QE systems was often similar across SMT and NMT datasets, suggesting a degree of robustness.
- New Error Types are Challenging: Predicting missing words and identifying source words that cause errors proved to be more difficult tasks, with fewer submissions and generally lower performance compared to predicting target word errors.
- Phrase-level Annotation Difficulty: The phrase-level task (Task 3a) showed a degradation in performance compared to word-level prediction (Task 2), possibly due to the indirect labeling of words based on phrase annotations.
- Document-level QE Potential: The document-level task, while new, showed promising results with Pearson correlation scores comparable to sentence-level prediction, indicating potential for future work in this area.
- Underutilization of Detailed Post-Editing Data: Surprisingly, no participating system requested the additional post-editing information (time, keystrokes, edits), leaving the utility of this data for QE an open question.
Key Results
- Task 1 (Sentence-level): The QEBrain and UNQE models achieved top performance on English-German and German-English tasks.
- Task 2 (Word-level): QEBrain also won for English-German and German-English. Performance on NMT data was generally lower than on SMT data for English-German.
- Task 3 (Phrase-level): The baseline system remained competitive, and the task proved challenging, especially for predicting incorrect word order within phrases.
- Task 4 (Document-level): The baseline system performed well, and the neural model SHEF-PT-indomain showed a modest improvement.
- Overall: The WMT 2018 QE shared task was the largest to date, with a wide range of systems and tasks, providing valuable insights into the state-of-the-art in automatic MT quality estimation.
Limitations and Future Work
- Dataset Size and Distribution: Differences in dataset sizes and HTER score distributions between SMT and NMT data made direct comparisons challenging.
- New Task Difficulty: The newly introduced tasks for predicting missing words and source word errors were more difficult and had fewer participants.
- Phrase-level Annotation: The indirect labeling of words from phrase annotations in Task 3a led to performance degradation.
- Utility of Detailed Post-Editing Data: The lack of system requests for detailed post-editing data means its utility for QE remains an open research question.
- Future Work: The paper suggests future research directions including improving performance with smaller datasets, better prediction of source-related errors, leveraging additional post-editing evidence, and exploring new deep learning architectures for document-level QE. CodaLab competitions have been set up for future benchmarking on blind test sets.
Summary updated