Translation Error Detection as Rationale Extraction

Marina Fomicheva, Lucia Specia, Nikolaos Aletras

Abstract

Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results in predicting the overall quality of translated sentences. However, detecting specifically which translated words are incorrect is a more challenging task, especially when dealing with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE; and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.

Introduction

In Machine Translation (MT), Quality Estimation (QE) aims to predict translation quality without a reference text. While sentence-level QE has seen significant progress, word-level QE—identifying specific erroneous tokens—remains difficult due to the high cost of obtaining human-annotated labels. This paper proposes a paradigm shift: instead of training supervised word-level models, we can treat word-level error detection as a rationale extraction task from existing sentence-level models.

Methodology: Rationale Extraction for QE

The core hypothesis is that a sentence-level model must "focus" on specific errors to determine a quality score. By applying feature attribution methods to a state-of-the-art sentence-level model (based on XLM-R), we can identify which words most influence the quality prediction.

Comparison between standard supervised word-level QE (left) and the proposed semi-supervised rationale extraction approach (right).

We evaluated four attribution methods:

  • Integrated Gradients: A gradient-based method tracing the path from a baseline to the input.
  • Information Bottleneck: A perturbation method injecting noise into hidden layers.
  • Attention: Averaging self-attention weights across transformer heads.
  • LIME: A model-agnostic local linear approximation.

Layer-wise Analysis and the "Sweet Spot"

A key finding of this research is that error detection performance is not uniform across the transformer architecture. By analyzing attributions across all hidden layers, we identified a "sweet spot" in the later layers (specifically layers 9-11).

Average attribution scores for Source, Target, and "Bad" tokens across transformer layers, showing a distinct peak for errors in later layers.

Early layers tend to focus on token identity, while the final layer is often overly specialized for the regression output. The middle-to-late layers are where the model has integrated sufficient contextual information to recognize that a specific word is a translation error. As shown in the figure above, the attribution for tokens labeled as "Bad" by humans significantly increases in these later layers, confirming that the model's internal representations are indeed sensitive to errors.

Results and Performance

The study utilized the MLQE-PE dataset across three language pairs: Romanian-English (Ro-En), Estonian-English (Et-En), and Nepalese-English (Ne-En). The results demonstrate that semi-supervised rationale extraction is highly effective:

  • Competitive Accuracy: For Ro-En, attention-based rationales achieved an Average Precision (AP) of 0.73, nearing the 0.81 AP of fully supervised state-of-the-art models despite using zero word-level training labels.
  • Robustness: Integrated Gradients and Attention proved to be the most consistent methods across different languages.
  • Interpretability Benchmark: Because translation errors are objective, QE serves as a rigorous benchmark for evaluating whether explainability methods align with human judgment.

Conclusion

This work demonstrates that sentence-level QE models naturally learn to identify translation errors as part of their quality assessment process. By extracting these rationales, we can perform word-level error detection without expensive supervised data. This approach not only provides a practical tool for MT quality assessment but also offers a new framework for evaluating the plausibility of model explanations in NLP.

Summary updated

Abstract

Recent Quality Estimation (QE) models based on multilingual pre-trained representations have achieved very competitive results in predicting the overall quality of translated sentences. However, detecting specifically which translated words are incorrect is a more challenging task, especially when dealing with limited amounts of training data. We hypothesize that, not unlike humans, successful QE models rely on translation errors to predict overall sentence quality. By exploring a set of feature attribution methods that assign relevance scores to the inputs to explain model predictions, we study the behaviour of state-of-the-art sentence-level QE models and show that explanations (i.e. rationales) extracted from these models can indeed be used to detect translation errors. We therefore (i) introduce a novel semi-supervised method for word-level QE; and (ii) propose to use the QE task as a new benchmark for evaluating the plausibility of feature attribution, i.e. how interpretable model explanations are to humans.
BibTeX
@inproceedings{fomicheva2022translation,
  author = {Marina Fomicheva and Lucia Specia and Nikolaos Aletras},
  title = {{Translation Error Detection as Rationale Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=bwtoJEC1pD},
}
Figure