Abstract
In this work, we investigate whether side information is helpful in neural machine translation (NMT). We study various kinds of side information, including topical information, personal trait, then propose different ways of incorporating them into the existing NMT models. Our experimental results show the benefits of side information in improving the NMT models.
Introduction
Standard Neural Machine Translation (NMT) models typically operate on a "text-in, text-out" basis, relying solely on the source sentence to generate a translation. However, language is inherently ambiguous; a word like "bank" could refer to a financial institution or a riverbank, and the correct translation often depends on the broader context or topic. While humans use surrounding context and metadata to resolve these ambiguities, most NMT systems ignore this "side information."
This paper explores how metadata—such as topic keywords, speaker traits, or document classifications—can be integrated into NMT architectures (specifically the Transformer) to help resolve lexical ambiguity, handle out-of-vocabulary terms, and maintain topic relevancy.
Methods for Incorporating Side Information
The researchers investigated four primary strategies for incorporating side information, represented as embedded bag-of-words vectors:
- Source Prefix/Suffix: Metadata is appended or prepended as text tokens to the source sentence. This requires no changes to the model architecture.
- Target Prefix: Metadata is added as a prefix to the target sequence. This allows the model to learn to generate the metadata itself if it is missing at test time.
- Output Layer Integration: The side information embedding is projected directly into the model's final output layer, just before the softmax transformation. This allows the metadata to directly influence the final lexical choice.
- Multi-task Learning (MTL): The model is trained to simultaneously translate the sentence and classify the side information (e.g., predicting topic keywords from the source encoder's representation).
Datasets and Side Information
The authors evaluated their methods across three distinct datasets, each providing different types of side information:
- TED Talks: Includes sentence-level topic keywords (e.g., "arts," "technology").
- Personalised Europarl: Includes metadata about the speaker's gender.
- Patent MT (PatTR): Includes International Patent Classification (IPC) codes, providing a hierarchical categorization of technology areas.
Experimental Results
The Output Layer method emerged as the most successful approach, consistently outperforming the baseline and other strategies.
Key Findings
- Topical vs. Personal Traits: Side information related to the topic (TED Talks or patent categories) proved significantly more beneficial than personal traits like gender. This suggests that topical context has a more direct impact on word choice.
- Performance Gains: On the TED Talks dataset, the Output Layer method achieved a statistically significant improvement of +0.51 BLEU over the baseline. On the PatTR dataset, it showed an improvement of +0.46 BLEU using detailed classification labels.
- Category-wise Analysis: The output-layer approach improved performance across almost all individual categories in the TED dataset, as shown in the figure below.
Interestingly, simple prefix/suffix methods sometimes performed worse than the baseline in the Transformer architecture. The authors speculate that the Transformer's powerful multi-head attention mechanism might get "confused" by extra tokens if they aren't integrated more structurally.
Conclusion
Integrating side information, particularly at the output layer, provides a robust way to improve NMT quality by leveraging topical context. Future work could explore more sophisticated Multi-task Learning architectures or hybrid approaches that combine output-layer integration with shared representations in the encoder. These techniques could potentially be extended to other NLP tasks such as summarization or reading comprehension.
Summary updated