Faithful Target Attribute Prediction in Neural Machine Translation

Xing Niu, Georgiana Dinu, Prashant Mathur, Anna CurreyarXiv 2021 · OpenReview

Abstract

The training data used in NMT is rarely controlled with respect to specific attributes, such as word casing or gender, which can cause errors in translations. We argue that predicting the target word and attributes simultaneously is an effective way to ensure that translations are more faithful to the training data distribution with respect to these attributes. Experimental results on two tasks, uppercased input translation and gender prediction, show that this strategy helps mirror the training data distribution in testing. It also facilitates data augmentation on the task of uppercased input translation.

Introduction

Standard Neural Machine Translation (NMT) models are highly effective at general translation but often struggle with specific attributes like word casing or gender. For instance, a model might fail to preserve the casing of a headline or exhibit gender bias by defaulting to masculine forms for certain professions. This issue stems from a lack of "faithfulness"—the degree to which a model's output distribution reflects the attribute distribution in its training data. Because most training corpora are inconsistently sourced, NMT models tend to amplify skews rather than accurately reflecting the data.

Methodology: Factored NMT

To address this, researchers leverage Factored NMT. Instead of predicting a single surface word, the model decomposes the target into multiple factors. For example, the word "NEURAL" is predicted as a base word factor ("neural") and a casing factor ("uppercase").

Using the Sockeye NMT toolkit, the researchers implemented independent output layers for these factors. The embeddings for each factor are summed and fed back into the decoder. This disentanglement allows the model to learn the semantic content of words separately from their stylistic or grammatical attributes, leading to more robust representations.

Experimental Setup

The approach was evaluated using several large-scale and specialized datasets:

  • ParaCrawl v7.1: For training EN-DE and EN-FR models.
  • WMT Newstest (2011–2020): For standard benchmarking.
  • Gender-Profession Dataset: A new set of 353 gender-distinct translation pairs for professions.
  • WinoMT: For in-context gender translation accuracy.

Results and Discussion

Faithfulness to Training Distribution

The study introduces metrics like the "Uppercasing Preservation Ratio" (UPR) to measure faithfulness. As shown in Figure 1, factored models (specifically those with target-side factors) align much more closely with the training data distribution across the entire spectrum of preservation ratios, whereas standard models deviate significantly.

Factored models (target/both) align closely with training distributions, whereas standard models (none) deviate significantly.

Robustness in Data Augmentation

A significant finding involves data augmentation (DA). In standard NMT, adding a small amount of augmented data (e.g., all-caps sentences) often degrades translation quality because the model's embeddings for rare tokens are poorly trained. Factored NMT mitigates this by sharing base word embeddings across all attribute styles.

As illustrated in Figure 6, the factored model ("both") maintains a stable BLEU score even when the DA ratio is extremely low ($2^{-5}\%$), whereas standard models suffer significant quality drops.

Factored models maintain high BLEU scores even with minimal augmented data, unlike standard models which suffer significant quality drops.

Gender Bias Mitigation

Factored models also showed improvements in gender accuracy. On the WinoMT benchmark, explicit prediction of gender markers helped the model better attend to context, improving F1 scores across masculine, feminine, and stereotypical subsets. The Mean Squared Error (MSE) of predicted gender ratios was reduced by 11% compared to baseline models.

Conclusion

Predicting target words and attributes simultaneously through Factored NMT enhances model faithfulness and robustness. This approach ensures that output distributions mirror training data and makes data augmentation significantly more effective for handling noisy or specialized inputs. Future work will explore sentence-level attributes and the handling of multiple overlapping grammatical features.

Summary updated

Abstract

The training data used in NMT is rarely controlled with respect to specific attributes, such as word casing or gender, which can cause errors in translations. We argue that predicting the target word and attributes simultaneously is an effective way to ensure that translations are more faithful to the training data distribution with respect to these attributes. Experimental results on two tasks, uppercased input translation and gender prediction, show that this strategy helps mirror the training data distribution in testing. It also facilitates data augmentation on the task of uppercased input translation.
BibTeX
@misc{niu2021faithful,
  author = {Xing Niu and Georgiana Dinu and Prashant Mathur and Anna Currey},
  title = {{Faithful Target Attribute Prediction in Neural Machine Translation}},
  year = {2021},
  eprint = {2109.12105},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.12105},
}
Figure