MIPT System for World-Level Quality Estimation

Mikhail Mosyagin, Varvara LogachevaACL 2019

TL;DR

This paper explores different neural network architectures for word-level quality estimation of machine translations, aiming to identify specific words in a translation that are incorrect. They propose a modified phrase-level Shef-bRNN model enhanced with features from BERT and transformer encoders, achieving competitive results on the WMT 19 English-German dataset.

What Problem Does This Paper Address?

Assessing the quality of machine translation (MT) is crucial, but traditional metrics like BLEU rely on comparing MT output to human-created reference translations. The WMT 19 conference introduced a shared task focused on "quality estimation" (QE) without relying on references. Specifically, this paper tackles the challenge of word-level quality estimation, where the goal is to pinpoint individual words in an MT output that are erroneous. This extends to identifying "gaps" (omitted words) and even labeling source words that led to these translation errors. This granular level of error detection is vital for understanding MT system weaknesses and for post-editing efforts.

Methods

The authors build upon existing architectures, primarily a phrase-level Shef-bRNN, and experiment with incorporating various feature extraction methods and sequence labeling techniques.

  • Base Model (Modified Shef-bRNN): They start with a phrase-level Shef-bRNN, which uses bidirectional LSTMs (bi-LSTMs) to encode source and target sentences. Global attention mechanisms are employed to create context vectors, summarizing relevant parts of the source sentence for each target word. They also introduce a "self-context" vector, capturing information about other target words.
  • Feature Augmentation:
    • Baseline Features: They integrate human-crafted features previously used in WMT 18, which include numerical (e.g., word counts) and categorical (e.g., part-of-speech tags) information.
  • BERT Features: They leverage a pre-trained BERT model (specifically, bert-base-multilingual-cased) to extract contextual embeddings for target words. Importantly, they do not fine-tune BERT, using it solely as a feature extractor. They concatenate the first and last WordPiece token outputs from BERT to form a fixed-size feature vector.
  • Transformer Encoder: They replace the bi-LSTM encoders with transformer encoders (Vaswani et al., 2017) to capture more contextual information through self-attention.
  • Sequence Labeling: For the final tag prediction (OK/BAD for words, or gap labels), they use Conditional Random Fields (CRFs) (Lafferty et al., 2001), a standard technique for sequence labeling tasks.
  • Gap Labeling: A novel approach is proposed for gap labeling. Instead of using a dummy word, they construct a new sequence of features by concatenating features from adjacent words and feeding this into a separate CRF model.
  • Source Word Labeling: Source word labels are derived from alignments between the source and post-edited target sentences. If a target word is labeled BAD, all aligned source words are also labeled BAD.

Datasets

  • WMT 19 English-German Dataset: This dataset was used for training and evaluating all models. It includes source sentences, machine translations, and gold-standard labels for target words, source words, and gaps.

Key Insights and Ideas

  • Feature Combination is Key: The most significant improvements come from combining different feature types. The RNN+Baseline+BERT and Transformer+Baseline+BERT models consistently outperform simpler configurations.
  • BERT's Contribution: While BERT alone doesn't revolutionize performance, its inclusion as an additional feature source provides a noticeable boost, particularly for target word prediction.
  • Transformer vs. RNN: Replacing bi-LSTMs with transformer encoders offers a slight improvement, suggesting that the self-attention mechanism in transformers is beneficial for capturing broader context.
  • Gap Labeling Strategy: The proposed method for gap labeling, which avoids increasing sequence length with dummy words, is an interesting adaptation to handle omitted words.
  • Baseline Features Still Relevant: The human-crafted baseline features, despite being older, still contribute positively to the overall performance, highlighting the value of diverse feature engineering.

Key Results

The paper presents results across three main prediction tasks: target words, source words, and gaps.

  • Target Prediction: The Transformer+Baseline+BERT model achieved the highest F1 Mult score of 0.33 on the test dataset, outperforming the RNN+Baseline+BERT (0.30). Both significantly surpassed the WMT 18 baseline (0.18).
  • Source Prediction: The RNN+Baseline+BERT model achieved an F1 Mult score of 0.30 on the test dataset, while the Transformer+Baseline+BERT achieved 0.27.
  • Gap Prediction: Both RNN+Baseline+BERT and Transformer+Baseline+BERT achieved an F1 Mult score of 0.14, showing strong performance in identifying gaps.
  • Comparison to Shef-bRNN: The authors note that their best models perform comparably to or slightly better than the word-level Shef-bRNN on development sets, with Shef-bRNN showing particularly strong performance on target prediction.

Limitations and Future Work

  • BERT Fine-tuning: The authors acknowledge that they did not fine-tune the BERT model, which could potentially lead to further performance gains.
  • Multi-task Learning: They mention that multi-task learning, as explored in other works (e.g., Kim et al., 2017), could be a promising direction for improvement.
  • Architecture Exploration: While they explored transformers, further investigation into different transformer configurations or other advanced architectures could yield better results.
  • Dataset Size: The performance of models on smaller datasets or for different language pairs is not discussed.

Summary updated

BibTeX
@inproceedings{mosyagin2019mipt,
  author = {Mikhail Mosyagin and Varvara Logacheva},
  title = {{MIPT System for World-Level Quality Estimation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure