Controlling Machine Translation for Multiple Attributes with Additive Interventions

Andrea Schioppa, David Vilar, Artem Sokolov, Katja FilippovaEMNLP 2021 · OpenReview

Abstract

Standard approaches to controlling attributes in Neural Machine Translation (NMT), such as length or politeness, typically rely on discrete source-side tags. However, these methods suffer from coarseness, interference between multiple tags, and inflexibility. This paper presents a novel approach using additive vector-valued interventions in the model's latent space. By adding learned perturbation vectors to the encoder's output, we enable continuous, fine-grained control over multiple attributes simultaneously. Our method, which can be efficiently fine-tuned onto pre-trained models, demonstrates superior precision in attribute control and improvements in translation quality without requiring full model retraining.

1. Introduction

Neural Machine Translation (NMT) has achieved high fluency, yet controlling specific stylistic or structural attributes—such as output length, politeness, or monotonicity—remains a challenge. The industry standard involves prepending discrete tags (e.g., <short>) to the source text. This approach is limited: continuous attributes must be binned into arbitrary categories, multiple tags can interfere with each other unpredictably, and adding new attributes often necessitates retraining the entire model. We propose a shift from these discrete "tags" to continuous "dials" via additive interventions in the latent space.

2. Methodology

The proposed method moves control from the input text to the model's internal representations. For each target attribute $a$, we define an intervention vector $V_a$. The total perturbation $V$ is calculated as a weighted sum: $V = \sum w_a V_a$, where $w_a$ serves as a continuous scalar "dial."

Architecture of the additive intervention model, showing the perturbation vector V being added to the encoder output before entering the decoder layers.

This perturbation is added to the output of the Transformer encoder. To maintain efficiency and stability, the intervention is "light-weight": only the last few layers of the decoder are modified to process this perturbed representation, while the rest of the model can remain frozen. This allows for fine-tuning existing high-quality models to support new attributes with minimal additional training.

3. Experimental Setup

The approach was evaluated on English-to-German (EN-DE) and English-to-Japanese (EN-JA) translation tasks using standard benchmarks:

  • WMT17: For general translation quality.
  • OpenSubtitles: For testing politeness and robustness.
  • JESC (Japanese-English Subtitle Corpus): Specifically for Japanese politeness levels.

4. Results and Discussion

Our experiments show that additive interventions provide significantly more precise control than tagging. In length control tasks, the "Add" model closely follows the ideal linear relationship between the intervention weight and the realized output length.

Performance of the additive intervention model in length control. (a) shows the precise linear control of output length compared to the ideal. (c) illustrates that BLEU scores peak at the natural translation length and degrade gracefully as the intervention is adjusted.

Key findings include:

  • Precision: The additive model avoids the "staircase" effect of discrete tagging, allowing for smooth adjustments.
  • Quality Preservation: Setting weights to zero recovers the base model's behavior. Furthermore, providing explicit attribute information (like politeness) can improve BLEU scores by resolving source-text ambiguities (+0.6 for German, +2.5 for Japanese).
  • Stability: Because vector addition is commutative, the order of multiple interventions does not affect the outcome, solving the interference problem seen with tags.
  • Efficiency: Control can be added to a pre-trained model in approximately 20% of the original training time.

5. Conclusion

Additive interventions offer a practical and powerful alternative to tagging for multi-attribute NMT control. While some attributes like monotonicity prove more difficult to learn through fine-tuning alone, the "dial" approach provides a scalable framework for fine-grained control. Future work will explore unsupervised learning of these vectors and extending the method to broader domain adaptation tasks.

Summary updated

BibTeX
@inproceedings{schioppa2021controlling,
  author = {Andrea Schioppa and David Vilar and Artem Sokolov and Katja Filippova},
  title = {{Controlling Machine Translation for Multiple Attributes with Additive Interventions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=74GKrHuMAa},
}
Figure