Sometimes We Want Translationese

Prasanna Parthasarathi, Koustuv Sinha, Joelle Pineau, Adina WilliamsarXiv 2021 · arXiv

Abstract

State-of-the-art Neural Machine Translation (NMT) systems are heavily optimized for robustness, enabling them to produce grammatical outputs even from noisy or ungrammatical inputs. However, this optimization often comes at the cost of faithfulness—the ability to preserve the unique structural characteristics of the source text. This paper explores this trade-off through the lens of word-order perturbations. By evaluating Helsinki/OPUS-MT models across seven languages, we demonstrate that current systems aggressively "hallucinate" corrected versions of source text, prioritizing normative grammar over structural fidelity.

Introduction

The primary goal of NMT research has long been the production of high-quality, fluent translations. To achieve this, models are trained to be robust against common errors like typos or grammatical slips. While beneficial for general use, this creates a "normative" bias where the model assumes any deviation from standard grammar is an error to be fixed. In contexts such as language tutoring or literary translation, preserving the original's non-standard structure is often more important than fluency. We lack quantitative metrics to evaluate when a model is being "too robust" at the expense of being faithful.

Methods

The authors propose a framework to quantify the robustness-faithfulness trade-off using 16 perturbation functions applied to English source sentences. These perturbations are categorized into random shuffles, Part-of-Speech (PoS) tag-based swaps, and dependency tree-based transformations.

Original vs. Mirrored dependency trees used for structural perturbations.

Two metrics are defined:

  1. Robustness ($\beta_1$): Measures how closely the translation of a perturbed sentence matches the gold-standard translation of the original sentence.
  2. Faithfulness ($\beta_2$): Measures how closely the translation matches a target sentence perturbed in the same manner as the source.

The study evaluates Helsinki/OPUS-MT models translating from English into German, French, Russian, Japanese, Chinese, Spanish, and Italian.

Results

The experimental results reveal a consistent trend: NMT models are fundamentally designed to "clean up" input. Across all languages and perturbation types, robustness scores ($\beta_1$) significantly outweighed faithfulness scores ($\beta_2$).

Robustness vs. Faithfulness scores across (a) target languages and (b) perturbation types.

Key findings include:

  • The Hallucination of Correctness: Models effectively "guess" the intended clean sentence rather than translating the actual jumbled input.
  • Syntactic Sensitivity: Models are most robust to PoS-based swaps where semantic clarity remains high, but struggle more with dependency tree rotations that disrupt core logical structures.
  • Language Variance: Robustness varies by target language; for instance, the Japanese model exhibited the highest robustness, potentially due to the language's flexible word order or specific model biases.

Conclusion

Current NMT systems act as aggressive "normative" filters. While this ensures fluency, it strips away "translationese"—the literal, structural traces of the source language that are sometimes desirable. Future research should focus on "controllable" NMT architectures that allow users to adjust the balance between robustness and faithfulness depending on the specific application, from technical documentation to creative arts.

Summary updated

Abstract

Rapid progress in Neural Machine Translation (NMT) systems over the last few years has been driven primarily towards improving translation quality, and as a secondary focus, improved robustness to input perturbations (e.g. spelling and grammatical mistakes). While performance and robustness are important objectives, by over-focusing on these, we risk overlooking other important properties. In this paper, we draw attention to the fact that for some applications, faithfulness to the original (input) text is important to preserve, even if it means introducing unusual language patterns in the (output) translation. We propose a simple, novel way to quantify whether an NMT system exhibits robustness and faithfulness, focusing on the case of word-order perturbations. We explore a suite of functions to perturb the word order of source sentences without deleting or injecting tokens, and measure the effects on the target side in terms of both robustness and faithfulness. Across several experimental conditions, we observe a strong tendency towards robustness rather than faithfulness. These results allow us to better understand the trade-off between faithfulness and robustness in NMT, and opens up the possibility of developing systems where users have more autonomy and control in selecting which property is best suited for their use case.
BibTeX
@misc{parthasarathi2021sometimes,
  author = {Prasanna Parthasarathi and Koustuv Sinha and Joelle Pineau and Adina Williams},
  title = {{Sometimes We Want Translationese}},
  year = {2021},
  eprint = {2104.07623},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.07623},
}
Figure