Input Combination Strategies for Multi-Source Transformer Decoder

Jindřich Libovický, Jindřich Helcl, David MarečekEMNLP 2018

TL;DR

This paper introduces four novel strategies for combining information from multiple input sources (like different languages or text and images) within the Transformer architecture for sequence-to-sequence tasks. These methods, including serial, parallel, flat, and hierarchical attention, significantly improve performance on multimodal and multi-source machine translation compared to single-source baselines.

What Problem Does This Paper Address?

Modern sequence-to-sequence models, particularly the Transformer architecture, have achieved remarkable success in tasks like machine translation. However, these models typically operate on a single input source. Many real-world scenarios, however, involve multiple sources of information. For instance, multimodal translation requires understanding both an image and its textual description to generate a translation, while multi-source machine translation aims to translate a sentence given its equivalents in several source languages. Existing methods for handling multiple sources were primarily developed for recurrent neural networks (RNNs) and are not directly applicable to the Transformer's encoder-decoder attention mechanism. This paper bridges this gap by proposing and evaluating new ways to integrate information from multiple encoders into a single Transformer decoder.

Methods

The core of this paper lies in proposing four distinct strategies for combining the outputs of multiple encoders before they are fed into the Transformer decoder's attention mechanism:

  • Serial: This approach processes each encoder's output sequentially. The output of the attention to one encoder becomes the input (query) for attending to the next encoder. This is a natural extension of the Transformer's sub-layer structure.
  • Parallel: Here, the decoder attends to each encoder independently and in parallel. The resulting context vectors from each attention are then summed up.
  • Flat: This strategy treats all encoder states from all sources as a single, flattened set of keys and values. The decoder then attends to this combined set, effectively modeling a joint distribution over all input tokens. This is an adaptation of a method used with RNNs.
  • Hierarchical: This method first computes attention independently for each source, and then treats these resulting context vectors as inputs to a second level of attention. This allows for a more structured combination of information.

The paper also details the standard Transformer decoder architecture, including multi-head scaled dot-product attention, self-attention, and feed-forward layers, all connected by residual connections.

Datasets

The paper evaluates its proposed methods on two benchmark datasets:

  • Multi30k: Used for multimodal machine translation (MMT). This dataset contains triplets of images, English captions, and their translations into German, French, and Czech.
  • Europarl: Used for multi-source machine translation (MSMT). The authors utilized Spanish, French, German, and English as source languages and Czech as the target language, creating a dataset of 511k sentence tuples for training.

Key Insights and Ideas

  • Transformer's Flexibility: The Transformer architecture, with its distinct sub-layers and residual connections, proves to be adaptable to various multi-source input combination strategies, going beyond simple extensions of existing RNN-based methods.
  • MMT Benefits: For multimodal translation, adding visual features significantly boosts performance, especially for English-to-French translation, suggesting that visual input provides crucial information for certain language pairs.
  • MSMT Gains: All proposed multi-source strategies outperform a single-source baseline in multi-source machine translation, demonstrating the value of leveraging multiple linguistic inputs.
  • Strategy Performance Varies: While the serial strategy was the best performer in MSMT, the flat combination excelled in MMT. This suggests that the optimal strategy can be task-dependent.
  • Robustness to Noise: The hierarchical attention strategy, particularly with modality gating, appears to make models more robust to noisy visual input in MMT.
  • English Dominance in MSMT: Adversarial evaluations for MSMT reveal that models heavily rely on the English source, using other languages for minor output refinements. This highlights a potential bias or reliance on the primary language.

Key Results

  • MMT: The proposed input combination strategies significantly surpassed the text-only baseline in English-to-French translation. Performance on other target languages showed slight improvements. Adversarial evaluation indicated that models utilize both visual and textual inputs, and incorrect visual input degrades performance.
  • MSMT: All proposed strategies outperformed the English-to-Czech single-source baseline. The serial strategy achieved the best results, followed closely by parallel and flat strategies. Adversarial evaluation showed that models heavily relied on English, with other sources contributing less significantly.

Limitations and Future Work

  • Flat Strategy Struggles: The flat combination strategy performed worse than others in MMT, potentially due to difficulties in learning a unified representation of diverse input modalities (text and image). The authors hypothesize that optimization might have failed to find a common representation.
  • English Reliance in MSMT: The adversarial evaluation in MSMT revealed a strong reliance on the English source language. Future work could explore methods to balance the influence of different source languages more effectively.
  • Task-Specific Optimization: The paper notes that the best-performing strategy can differ between MMT and MSMT. Further research could investigate how to dynamically select or adapt combination strategies based on the task or input characteristics.
  • Adversarial Evaluation Insights: The adversarial evaluations provided valuable insights into model behavior, particularly the reliance on specific sources. This approach could be further utilized to understand and improve model robustness.

Summary updated

BibTeX
@inproceedings{libovicky2018input,
  author = {Jindřich Libovický and Jindřich Helcl and David Mareček},
  title = {{Input Combination Strategies for Multi-Source Transformer Decoder}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure