Learning to Detect Unacceptable Machine Translations for Downstream Tasks

Meng Zhang, Xin Jiang, Yang Liu, Qun LiuarXiv 2020 · arXiv

Introduction

Traditional machine translation (MT) evaluation metrics, such as BLEU or HTER, often fail to capture the practical utility of a translation within a larger automated pipeline. A translation might be grammatically imperfect yet still allow a downstream task, such as sentiment analysis, to function correctly. Conversely, a single mistranslated word in a technical or legal context can be catastrophic despite a high BLEU score. This paper proposes a shift from general quality estimation to task-specific "acceptability."

Methodology

Defining Acceptability

The authors define a translation $t$ as acceptable for a specific downstream task $f$ if the output of the task on the translation matches the output on the gold-standard reference $r$. Formally, this is expressed as $f(t) = f(r)$. This definition allows for a binary classification of translation quality based on functional consistency rather than linguistic perfection.

Automatic Annotation

A key contribution of this work is the method for generating large-scale training data without human intervention. By using existing parallel corpora (source $s$, reference $r$), the authors generate translations $t$ using an MT system. They then apply a downstream NLP tool (e.g., a sentiment classifier or NER tagger) to both $t$ and $r$. If the outputs are identical, the pair $(s, t)$ is labeled as "acceptable"; otherwise, it is "unacceptable."

Automatic annotation of translation acceptability using parallel corpora and downstream task outputs.

Detection Models

To predict acceptability at inference time (where the reference $r$ is unavailable), the authors developed a neural architecture called BiRNN. This model employs separate Bidirectional GRU layers to encode the source and target sentences. An attention mechanism integrates these representations into a single vector, which is then processed through non-linear layers to output a probability of acceptability.

Experimental Results

The framework was evaluated across three downstream tasks: subjectivity classification, sentiment analysis, and Named Entity Recognition (NER). The experiments utilized WMT18 Chinese-English data to train MT systems of varying quality levels.

Performance Comparison

The BiRNN model consistently outperformed both the baseline (which assumes all translations are acceptable) and a feature-based model (BiQuEst). The gains were particularly pronounced in the NER task, which proved to be highly sensitive to translation errors. For the weakest MT system tested, BiRNN improved detection accuracy for NER by approximately 25% absolute over the baseline.

Performance comparison of acceptability detection models across various tasks and MT systems.

Data Scaling and Task Sensitivity

The study found that the performance of the acceptability detector scales well with data, showing continued improvements up to 5 million training instances. Furthermore, the results highlighted that different tasks have different tolerance levels for MT errors. While sentiment analysis is relatively robust to minor translation flaws, NER acts as a "canary in the coal mine," frequently failing when even minor entities are mistranslated.

Conclusion

By reframing MT quality as task-specific acceptability, this research provides a scalable and automated way to build quality detectors. This approach leverages the abundance of parallel data to create models that are aware of the requirements of downstream applications, offering a more meaningful metric for real-world NLP pipelines. Future work may involve integrating these detectors directly into the MT training process to optimize for downstream success.

Summary updated

Abstract

The field of machine translation has progressed tremendously in recent years. Even though the translation quality has improved significantly, current systems are still unable to produce uniformly acceptable machine translations for the variety of possible use cases. In this work, we put machine translation in a cross-lingual pipeline and introduce downstream tasks to define task-specific acceptability of machine translations. This allows us to leverage parallel data to automatically generate acceptability annotations on a large scale, which in turn help to learn acceptability detectors for the downstream tasks. We conduct experiments to demonstrate the effectiveness of our framework for a range of downstream tasks and translation models.
BibTeX
@misc{zhang2020learning,
  author = {Meng Zhang and Xin Jiang and Yang Liu and Qun Liu},
  title = {{Learning to Detect Unacceptable Machine Translations for Downstream Tasks}},
  year = {2020},
  eprint = {2005.03925},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.03925},
}
Figure