Abstract
Standard approaches to controlling attributes in Neural Machine Translation (NMT), such as length or politeness, typically rely on discrete source-side tags. However, these methods suffer from coarseness, interference between multiple tags, and inflexibility. This paper presents a novel approach using additive vector-valued interventions in the model's latent space. By adding learned perturbation vectors to the encoder's output, we enable continuous, fine-grained control over multiple attributes simultaneously. Our method, which can be efficiently fine-tuned onto pre-trained models, demonstrates superior precision in attribute control and improvements in translation quality without requiring full model retraining.
1. Introduction
Neural Machine Translation (NMT) has achieved high fluency, yet controlling specific stylistic or structural attributes—such as output length, politeness, or monotonicity—remains a challenge. The industry standard involves prepending discrete tags (e.g., <short>) to the source text. This approach is limited: continuous attributes must be binned into arbitrary categories, multiple tags can interfere with each other unpredictably, and adding new attributes often necessitates retraining the entire model. We propose a shift from these discrete "tags" to continuous "dials" via additive interventions in the latent space.
2. Methodology
The proposed method moves control from the input text to the model's internal representations. For each target attribute $a$, we define an intervention vector $V_a$. The total perturbation $V$ is calculated as a weighted sum: $V = \sum w_a V_a$, where $w_a$ serves as a continuous scalar "dial."
This perturbation is added to the output of the Transformer encoder. To maintain efficiency and stability, the intervention is "light-weight": only the last few layers of the decoder are modified to process this perturbed representation, while the rest of the model can remain frozen. This allows for fine-tuning existing high-quality models to support new attributes with minimal additional training.
3. Experimental Setup
The approach was evaluated on English-to-German (EN-DE) and English-to-Japanese (EN-JA) translation tasks using standard benchmarks:
- WMT17: For general translation quality.
- OpenSubtitles: For testing politeness and robustness.
- JESC (Japanese-English Subtitle Corpus): Specifically for Japanese politeness levels.
4. Results and Discussion
Our experiments show that additive interventions provide significantly more precise control than tagging. In length control tasks, the "Add" model closely follows the ideal linear relationship between the intervention weight and the realized output length.
Key findings include:
- Precision: The additive model avoids the "staircase" effect of discrete tagging, allowing for smooth adjustments.
- Quality Preservation: Setting weights to zero recovers the base model's behavior. Furthermore, providing explicit attribute information (like politeness) can improve BLEU scores by resolving source-text ambiguities (+0.6 for German, +2.5 for Japanese).
- Stability: Because vector addition is commutative, the order of multiple interventions does not affect the outcome, solving the interference problem seen with tags.
- Efficiency: Control can be added to a pre-trained model in approximately 20% of the original training time.
5. Conclusion
Additive interventions offer a practical and powerful alternative to tagging for multi-attribute NMT control. While some attributes like monotonicity prove more difficult to learn through fine-tuning alone, the "dial" approach provides a scalable framework for fine-grained control. Future work will explore unsupervised learning of these vectors and extending the method to broader domain adaptation tasks.
Summary updated