Adaptive Transformers for Learning Multimodal Representations

Prajjwal Bhargava

TL;DR

This paper explores adaptive mechanisms for Transformers in multimodal tasks, aiming to improve computational efficiency and model interpretability. By introducing adaptive attention spans and sparse attention, the authors show these methods can help understand how models process complex multimodal inputs and potentially reduce computational costs without significant performance degradation.

What Problem Does This Paper Address?

Transformers have become a dominant architecture for multimodal learning, particularly in tasks combining vision and language. However, these models are often "over-parametrized," meaning they have a vast number of parameters that require significant computational resources and long training times. This makes them inaccessible for many researchers and contributes to a large carbon footprint. The paper addresses this by investigating adaptive methods that allow the model to dynamically adjust its behavior based on the input's complexity, rather than relying on fixed, computationally intensive operations. This aims to make multimodal Transformers more efficient and interpretable.

Methods

The core of this work lies in extending and applying adaptive mechanisms to Transformer architectures for multimodal tasks. The key techniques introduced or explored are:

  • Adaptive Attention Span: Instead of a fixed context window for attention heads, this method allows each head to learn an optimal span. This is achieved by introducing a masking function that dynamically adjusts the attention range based on the input sequence's complexity. This can lead to reduced computation by focusing attention on relevant parts of the input.
  • Adaptive Sparse Attention: This approach uses the "entmax" function, which can produce sparse attention distributions, unlike the standard softmax. Sparsity can lead to more efficient computation and potentially better interpretability by highlighting the most important relationships. The degree of sparsity is controlled by a learnable parameter, alpha.
  • LayerDrop: While not entirely novel, the paper investigates LayerDrop as a regularization technique. This method randomly drops entire layers during training, effectively creating a shallower network. The authors explore different configurations and drop rates to understand its impact on performance and efficiency.

Datasets

The primary dataset used for evaluation in this work is VQA 2.0 (Visual Question Answering). This dataset consists of images paired with questions about them, requiring the model to predict an answer from a set of choices. The paper mentions using pre-trained weights from LXMERT, which was likely trained on large-scale multimodal datasets, but the specific details of that pre-training are not the focus of this paper.

Key Insights and Ideas

  • Attention Spans Reveal Layer Sensitivity: The adaptive attention span mechanism provides insights into which layers are more sensitive to input complexity. Intermediate layers in multimodal encoders tend to require larger spans, suggesting they are crucial for integrating information from different modalities.
  • Vision Modality Perceived as More Complex: The study suggests that visual sequences are perceived as more complex than language inputs by the cross-modality encoder, requiring higher attention spans.
  • Sparsity Preferences Vary by Modality: Language modalities seem to benefit from sparse attention distributions, while vision modalities might converge towards denser mappings. When both modalities are involved, denser distributions are preferred, possibly due to the perceived complexity of vision.
  • LayerDrop's Trade-offs: While LayerDrop can reduce computational cost by creating shallower effective networks, it requires significantly more training time for convergence. Furthermore, aggressive LayerDrop can hinder performance compared to other adaptive methods.
  • Adaptive Methods for Efficiency: The adaptive approaches show promise in achieving performance comparable to standard attention mechanisms while being computationally more efficient, making them more accessible.

Key Results

  • VQA Performance: Table 1 shows that LXMERT with adaptive attention span and adaptive sparse attention achieves competitive results on the VQA dataset, with adaptive attention span slightly outperforming the baseline softmax LXMERT on test-dev and test-std. Adaptive sparse attention also shows comparable performance.
  • LayerDrop Impact: Table 1 indicates that LayerDrop (with a 10-6-6 configuration) leads to a significant drop in performance (around 66.4% accuracy) compared to the baseline (around 72.4%). Table 2 further shows that a shallower network (10-6-6) with LayerDrop performs better than a deeper one (9-5-5) with LayerDrop, suggesting a threshold for its effectiveness.
  • Computational Efficiency: The paper emphasizes that adaptive methods can significantly reduce computational costs, although specific FLOPS reduction numbers are not detailed for all methods. Adaptive attention span is noted to enable a reduction in FLOPS.

Limitations and Future Work

  • LayerDrop Training Time: LayerDrop requires substantially more training time for convergence, even though it can lead to shallower effective networks.
  • LayerDrop Performance Degradation: Aggressive LayerDrop can lead to a noticeable drop in performance, suggesting a trade-off between efficiency and accuracy.
  • Future Work: The authors suggest further exploration of adaptive and sparse mechanisms for even greater FLOPS and parameter reduction with minimal performance loss. They also highlight the need for more research into how these adaptive mechanisms can be further leveraged for interpretability. The paper also notes that the empirical results are encouraging, but more extensive future work is needed.

Summary updated

Abstract

The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. In this work, we extend adaptive approaches to learn more about model interpretability and computational efficiency. Specifically, we study attention spans, sparse, and structured dropout methods to help understand how their attention mechanism extends for vision and language tasks. We further show that these approaches can help us learn more about how the network perceives the complexity of input sequences, sparsity preferences for different modalities, and other related phenomena.
BibTeX
@inproceedings{bhargava2020adaptive,
  author = {Prajjwal Bhargava},
  title = {{Adaptive Transformers for Learning Multimodal Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=pvzdMdP8_uh},
}
Figure