Abstract
State-of-the-art Neural Machine Translation (NMT) systems are heavily optimized for robustness, enabling them to produce grammatical outputs even from noisy or ungrammatical inputs. However, this optimization often comes at the cost of faithfulness—the ability to preserve the unique structural characteristics of the source text. This paper explores this trade-off through the lens of word-order perturbations. By evaluating Helsinki/OPUS-MT models across seven languages, we demonstrate that current systems aggressively "hallucinate" corrected versions of source text, prioritizing normative grammar over structural fidelity.
Introduction
The primary goal of NMT research has long been the production of high-quality, fluent translations. To achieve this, models are trained to be robust against common errors like typos or grammatical slips. While beneficial for general use, this creates a "normative" bias where the model assumes any deviation from standard grammar is an error to be fixed. In contexts such as language tutoring or literary translation, preserving the original's non-standard structure is often more important than fluency. We lack quantitative metrics to evaluate when a model is being "too robust" at the expense of being faithful.
Methods
The authors propose a framework to quantify the robustness-faithfulness trade-off using 16 perturbation functions applied to English source sentences. These perturbations are categorized into random shuffles, Part-of-Speech (PoS) tag-based swaps, and dependency tree-based transformations.
Two metrics are defined:
- Robustness ($\beta_1$): Measures how closely the translation of a perturbed sentence matches the gold-standard translation of the original sentence.
- Faithfulness ($\beta_2$): Measures how closely the translation matches a target sentence perturbed in the same manner as the source.
The study evaluates Helsinki/OPUS-MT models translating from English into German, French, Russian, Japanese, Chinese, Spanish, and Italian.
Results
The experimental results reveal a consistent trend: NMT models are fundamentally designed to "clean up" input. Across all languages and perturbation types, robustness scores ($\beta_1$) significantly outweighed faithfulness scores ($\beta_2$).
Key findings include:
- The Hallucination of Correctness: Models effectively "guess" the intended clean sentence rather than translating the actual jumbled input.
- Syntactic Sensitivity: Models are most robust to PoS-based swaps where semantic clarity remains high, but struggle more with dependency tree rotations that disrupt core logical structures.
- Language Variance: Robustness varies by target language; for instance, the Japanese model exhibited the highest robustness, potentially due to the language's flexible word order or specific model biases.
Conclusion
Current NMT systems act as aggressive "normative" filters. While this ensures fluency, it strips away "translationese"—the literal, structural traces of the source language that are sometimes desirable. Future research should focus on "controllable" NMT architectures that allow users to adjust the balance between robustness and faithfulness depending on the specific application, from technical documentation to creative arts.
Summary updated