Adaptive Transformers for Learning Multimodal Representations

Prajjwal Bhargava

TL;DR

Multimodal transformers like LXMERT achieve impressive results in vision-and-language tasks, but they are notoriously compute-heavy and over-parameterized. This paper explores whether adaptive techniques—specifically adaptive attention spans, $\alpha$-entmax sparse attention, and LayerDrop—can retain competitive accuracy while reducing computation. Beyond efficiency gains, these mechanisms provide valuable interpretability insights, revealing that transformers naturally favor sparse attention for text but demand denser attention and broader context spans for visual and cross-modal representations.

What Problem Does This Paper Address?

Modern visiolinguistic architectures (e.g., LXMERT, ViLBERT, UNITER) require weeks of compute across massive GPU clusters to pre-train, incurring steep financial and environmental costs. Standard self-attention compounds this problem:

  1. Fixed context spans: Standard attention heads assume every token needs the same context window, regardless of sequence complexity.
  2. Dense distribution overhead: Softmax cannot assign true zero weights, forcing the network to spend computational capacity tracking irrelevant tokens.
  3. Rigid pruning methods: Techniques like static distillation or structural pruning enforce permanent architectural modifications rather than adapting dynamically to input complexity.

The author investigates how adaptive techniques—originally developed for text-only settings—perform when transferred to multimodal architectures, and what they reveal about cross-modal feature processing.

Methods

The paper integrates three adaptive techniques into LXMERT, a dual-stream transformer featuring 9 language layers, 5 vision layers, and 5 cross-modality encoder layers:

  • Adaptive Attention Span: Uses a learnable masking parameter $z$ per head combined with a ramp function $m_z(x) = \min[\max[\frac{1}{R}(R + z - x), 0], 1]$ to dynamically restrict the head's context window ($R$ controls softness).
  • Adaptive Sparse Attention ($\alpha$-entmax): Replaces standard softmax with $\alpha$-entmax, where the scalar curvature parameter $\alpha \in [1, 2]$ is learned per head. When $\alpha = 1$, it behaves like softmax; when $\alpha = 2$, it behaves like sparsemax, enabling exact zeros in attention matrices.
  • LayerDrop: A structured dropout method that probabilistically drops entire transformer layers during training (using an "Every Other" strategy, dropping $p=1$ layer per block), facilitating inference-time layer pruning.

Models were fine-tuned on a single NVIDIA P100 GPU using the LAMB optimizer wrapped inside Lookahead, managed by a cyclical learning rate schedule.

Datasets

The methods are evaluated on VQA 2.0 (Visual Question Answering), requiring the model to select an answer from 3,129 candidates given an image and question.

  • Train split: ~83k images, 444k questions.
  • Validation split: ~41k images, 214k questions.
  • Test split: ~81k images, 448k questions (split into test-dev and test-std).

Key Insights and Ideas

  • Visual features demand denser attention than language: Language encoder heads consistently favored sparse distributions ($\alpha \to 2$). Conversely, when handling visual tokens or cross-modal interactions, heads converged toward dense mappings ($\alpha < 1.5$), suggesting that visual tokens contain dispersed, non-redundant spatial information that resists hard pruning.
  • Cross-modal reasoning expands context requirements: As training progressed, self-attention spans for single modalities monotonically decreased (specializing into narrower windows). However, cross-modality encoder layers—particularly when reconciling vision and text together—maintained or expanded large attention spans.
  • Sparse attention sharpens prediction contrast: Qualitative analysis showed that $\alpha$-entmax produced sharper, high-confidence predictions on target classes compared to the diffuse confidences produced by softmax under adaptive attention spans.
  • Adaptive span and entmax conflict: Combining both adaptive attention spans and $\alpha$-entmax degraded performance significantly. Soft masking functions designed for softmax do not compose smoothly with hard sparse mappings.

Key Results

Evaluating on VQA 2.0:

Model / Configuration test-dev Acc (%) test-std Acc (%)
LXMERT baseline (w/ softmax) 72.42 72.54
LXMERT w/ Adaptive Sparse ($\alpha$-entmax) 71.73 71.97
LXMERT w/ Adaptive Attention Span 71.62 71.72
LXMERT w/ LayerDrop (10-6-6, $p=1$) 66.40 66.72
LXMERT w/ Span + Entmax (Ablation) 63.07 63.33

Both adaptive attention spans and adaptive sparse attention retained performance within ~0.6–0.8% of the full softmax baseline while operating with improved efficiency.

In contrast, LayerDrop performed poorly, lagging behind by ~5.7% accuracy while requiring roughly $3.5\times$ more training runtime to converge, indicating that coarse layer-skipping disrupts the fragile representations needed for joint vision-language reasoning.

Limitations and Future Work

  • No hyperparameter tuning: Results were obtained without exhaustive tuning of masking hyperparameter $R$ or optimizer schedules, leaving potential performance on the table.
  • LayerDrop inefficiency: LayerDrop failed to serve as an effective adaptive depth tool for multimodal setups due to slow convergence and steep accuracy drops.
  • Lack of hardware-level speedup metrics: Although adaptive spans and sparsity reduce theoretical FLOPs, the paper does not quantify actual wall-clock inference speedups or memory reduction on low-resource hardware.
  • Future directions: Developing synergistic mechanisms that can combine sparse attention and span reductions without destructive interference, and designing multimodal-specific structured dropout strategies.

Summary updated

Abstract

The usage of transformers has grown from learning about language semantics to forming meaningful visiolinguistic representations. These architectures are often over-parametrized, requiring large amounts of computation. In this work, we extend adaptive approaches to learn more about model interpretability and computational efficiency. Specifically, we study attention spans, sparse, and structured dropout methods to help understand how their attention mechanism extends for vision and language tasks. We further show that these approaches can help us learn more about how the network perceives the complexity of input sequences, sparsity preferences for different modalities, and other related phenomena.
BibTeX
@inproceedings{bhargava2020adaptive,
  author = {Prajjwal Bhargava},
  title = {{Adaptive Transformers for Learning Multimodal Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=pvzdMdP8_uh},
}

Loading related content...

Figure