Translation Error Detection as Rationale Extraction

Marina Fomicheva, Lucia Specia, Nikolaos Aletras

Abstract

Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results when predicting the overall quality of translated sentences. Predicting translation errors, i.e. detecting specifically which words are incorrect, is a more challenging task, especially with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.

Introduction

In Machine Translation (MT), Quality Estimation (QE) aims to predict translation quality without human references. While sentence-level models, leveraging large pre-trained transformers like XLM-R, have reached high accuracy, word-level error detection remains difficult. The primary bottleneck is the scarcity of human-annotated word-level data, which requires expensive bilingual expertise.

This research investigates whether sentence-level QE models inherently identify specific errors to arrive at their global quality scores. By "interrogating" these models through explainability techniques, we can extract rationales that highlight problematic words. This approach enables word-level error detection using only sentence-level labels, offering a more scalable solution for MT quality assessment.

Methodology

The core approach treats word-level error detection as a rationale extraction task. We utilize a pre-trained sentence-level QE model and apply post-hoc explainability methods to identify input features that most significantly influence the predicted score.

The proposed framework for extracting word-level rationales from a sentence-level QE model.

Four primary attribution techniques were evaluated:

  • Integrated Gradients: Attributes output to input features by integrating gradients along a path from a baseline.
  • Information Bottleneck: Injects noise into intermediate layers to identify essential representations.
  • Attention: Averages attention weights across Transformer layers.
  • LIME: Fits a local interpretable model around specific predictions.

Attribution was performed across different hidden layers to identify where the error-detecting signal is most prominent. The study used the MLQE-PE dataset across three language pairs: Romanian-English (high-resource), Estonian-English (mid-resource), and Nepalese-English (low-resource).

Results and Insights

The experiments confirm that the features identified as "relevant" by the model for predicting low quality align closely with actual translation errors. This suggests that QE models naturally learn to act as proofreaders.

Layer-wise attribution scores for different word categories, showing a peak for "Bad" words in middle-to-late layers.

A key finding is the existence of a "sweet spot" in the model architecture. Error detection performance peaks in the middle-to-late layers (layers 9-11). Early layers appear to focus on raw word identity, while the final layers are overly specialized for the regression task.

Quantitatively, the semi-supervised approach performed competitively with fully supervised models. For Romanian-English, Integrated Gradients achieved an AUC of 0.79. Furthermore, the Accuracy@Top-1 reached up to 0.91 for Nepalese-English, indicating that the word flagged as most important by the model is almost always a human-verified error.

Discussion and Conclusion

This work demonstrates that sentence-level QE models possess zero-shot capabilities for word-level error detection. Beyond QE, this task serves as an excellent benchmark for Explainable AI (XAI), as translation errors provide objective ground truth for evaluating the plausibility of model explanations.

Future research should address the current limitations, such as the dependency on sentence-level accuracy and the lack of distinction between error severities. Refining attribution baselines and tuning models for better rationale production could further bridge the gap between semi-supervised and supervised performance.

Summary updated

Abstract

Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results when predicting the overall quality of translated sentences. Predicting translation errors, i.e. detecting specifically which words are incorrect, is a more challenging task, especially with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.
BibTeX
@inproceedings{fomicheva2022translation,
  author = {Marina Fomicheva and Lucia Specia and Nikolaos Aletras},
  title = {{Translation Error Detection as Rationale Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=bwtoJEC1pD},
}
Figure