Multi-Task Neural Models for Translating Between Styles Within and Across Languages

Xing Niu, Sudha Rao, Marine Carpuat

Abstract: Xing Niu, Sudha Rao, Marine Carpuat. Proceedings of the 27th International Conference on Computational Linguistics.

2018.

Introduction

Language communication involves more than just the transmission of semantic content; it also encompasses stylistic nuances such as formality. While Neural Machine Translation (NMT) has achieved significant milestones in preserving meaning across languages, it often fails to capture these stylistic variations. For instance, translating a professional email requires a formal tone, whereas a text message necessitates a casual one.

A primary obstacle in developing style-sensitive translation models is the scarcity of data. Ideally, training such a model would require triplets consisting of a source sentence (e.g., in French) and two English translations: one formal and one informal. However, such parallel corpora are virtually non-existent. This research addresses the problem by leveraging Multi-Task Learning (MTL) to share knowledge between Formality Transfer (FT) within a single language and Formality-Sensitive Machine Translation (FSMT) across languages.

Methodology: The Multi-Task Architecture

The authors propose a multi-task neural model based on an attentional recurrent encoder-decoder architecture. The model is designed to handle two distinct but related tasks: Bi-Directional Formality Transfer (Bi-FT) and Formality-Sensitive Machine Translation (FSMT).

Overview of the multi-task neural model architecture for Bi-FT and FSMT.

Key Innovations

  1. Bi-Directional Formality Transfer: Instead of training separate models for informal-to-formal and formal-to-informal tasks, the researchers utilize a single model. They employ "side constraints" by prepending a specific tag—<F> for formal or <I> for informal—to the source sentence, directing the model toward the desired output style.
  2. Cross-Entropy Difference (CED) for Data Selection: To compensate for the lack of style-labeled French-English data, the authors used CED to rank a massive pool of general translation data against known formal/informal corpora. This allowed them to "pseudo-label" the most stylistically distinct translation pairs for training.
  3. Shared Representations: By training the model to map both French and English inputs to a target English output using a shared encoder and decoder, the model learns a unified representation of formality that transcends language boundaries.

Experimental Setup and Datasets

The model was trained and evaluated using two primary datasets:

  • GYAFC (Grammarly’s Yahoo Answers Formality Corpus): A dataset for Formality Transfer containing 110k pairs of informal sentences and human-written formal rewrites.
  • OpenSubtitles2016: A large dataset of movie and TV subtitles used for French-English translation. The authors filtered this to 2.5 million high-quality pairs using semantic similarity detection.

Results and Discussion

The multi-task approach yielded several key insights. Most notably, the model successfully learned to control the formality of French-to-English translations purely by learning from English-to-English style transfer data. The shared decoder internalizes that the <F> tag implies complex vocabulary and complete sentence structures, regardless of the source language.

Impact of parallel data size on formality transfer (a) and machine translation (b) performance.

As shown in the experimental results, adding translation data significantly improved performance on the English-only formality transfer task. This suggests that the translation data acts as a powerful regularizer, enhancing the model's overall English language modeling capabilities. The model achieved state-of-the-art BLEU scores for formality transfer at the time of publication.

Human evaluation further confirmed these findings, with the model maintaining high scores for meaning preservation (averaging ~2.9 out of 3.0). While the model occasionally struggled with informal conventions like all-caps text or slight meaning shifts (e.g., changing "women" to "girls" to sound more informal), it demonstrated robust style control across both tasks.

Conclusion

This work demonstrates that multi-task learning is an effective strategy for overcoming data scarcity in stylistic language tasks. By treating style as a universal constraint that can be shared across languages, the researchers developed a model capable of sophisticated formality control in both monolingual rewriting and bilingual translation. Future work may explore more complex architectures to further minimize meaning loss during style transformation and extend these methods to other stylistic dimensions.

Summary updated

BibTeX
@inproceedings{niu2018multi,
  author = {Xing Niu and Sudha Rao and Marine Carpuat},
  title = {{Multi-Task Neural Models for Translating Between Styles Within and Across Languages}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=xGsxXbb61hI},
}
Figure