Bilinear Attention Networks

Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang

TL;DR

Bilinear Attention Networks (BAN) improve multimodal learning by modeling the dense interactions between every pair of input channels—such as every word in a question and every region in an image. By leveraging low-rank bilinear pooling and a novel multimodal residual learning scheme, BAN achieves state-of-the-art results on VQA 2.0 and Flickr30k Entities while remaining computationally efficient.

What Problem Does This Paper Address?

In multimodal tasks like Visual Question Answering (VQA), models must "ground" language in visual data. Traditional attention mechanisms usually take one of two forms:

  1. Unitary Attention: One modality (e.g., a question vector) is used to attend to the other (e.g., image regions).
  2. Co-attention: Both modalities attend to each other, but usually independently or sequentially to keep computational costs low.

The problem is that these methods often neglect the fine-grained interactions between specific words and specific image regions. For a question like "What color are the man's pants?", a model should ideally look at the word "pants" while simultaneously attending to the corresponding region in the image. Modeling all these pairwise interactions ($N$ words $\times$ $M$ regions) is traditionally too expensive, leading researchers to use simplified, less effective summaries of the input.

Methods

The authors introduce Bilinear Attention Networks (BAN) to capture these dense interactions efficiently.

1. Bilinear Attention Map

Instead of a single attention vector, BAN computes a bilinear attention map $A \in \mathbb{R}^{\rho \times \phi}$ (where $\rho$ is the number of words and $\phi$ is the number of image regions). To make this computationally feasible, they use low-rank bilinear pooling. This technique approximates the interaction between two high-dimensional vectors by projecting them into a lower-dimensional space and using a Hadamard (element-wise) product.

2. Multimodal Residual Networks (MRN)

To further boost performance, the authors stack multiple "glimpses" of attention. Unlike previous methods that simply concatenate these glimpses, BAN uses a residual learning approach: $$f_{i+1} = \text{BAN}_i(f_i, Y; A_i) \cdot \mathbf{1}^T + f_i$$ This allows the model to refine its joint representation across up to eight glimpses without the parameter explosion or optimization difficulties seen in earlier architectures.

Figure 1: Overview of two-glimpse BAN. Two multi-channel inputs, φ-object detection features and ρ-length GRU hidden vectors, are used to get bilinear attention maps and joint representations to be used by a classifier. For the definition of the BAN, see the text in Section 3.

Datasets

The model was evaluated on two major benchmarks:

  • VQA 2.0: A large-scale visual question answering dataset designed to minimize language bias by providing different answers for the same question across different images.
  • Flickr30k Entities: A dataset for phrase localization (visual grounding), where the goal is to map specific phrases in a sentence to bounding boxes in an image.

Key Insights and Ideas

  • Dense Interaction is Key: Modeling the full matrix of word-region interactions provides a much richer signal than attending to an image based on a single, compressed question vector.
  • Efficiency through Factorization: You don't need a full-rank weight matrix to capture multimodal interactions. Low-rank factorization allows the model to consider every pair of channels with the same time complexity as standard co-attention.
  • Residuals for Attention: The shift from concatenation to residual summation for multi-glimpse attention is crucial. It makes the model more robust and allows for deeper attention stacks.
  • Robustness to Overfitting: As shown in the learning curves, bilinear attention is significantly more robust to overfitting than unitary or co-attention methods.
Figure 2: (a) learning curves. Bilinear attention (bi-att) is more robust to overfitting than unitary attention (uni-att) and co-attention (co-att). (b) validation scores for the number of parameters. The error bar indicates the standard deviation among three random initialized models, although it is too small to be noticed for over-15M parameters. (c) ablation study for the first-N-glimpses (x-axis) used in evaluation. (d) the information entropy (y-axis) for each attention map in four-glimpse BAN. The entropy of multiple attention maps is converged to certain levels.

Key Results

  • State-of-the-Art Performance: On VQA 2.0, BAN achieved a score of 70.35% (single model) and 71.84% (ensemble), outperforming the 2017 challenge winners.
  • Phrase Localization: On Flickr30k Entities, BAN reached 69.69% Recall@1, a massive 4.48% improvement over the previous state-of-the-art.
  • Inference Speed: Because BAN can process multiple entities in a sentence simultaneously through its multi-channel input, it achieved a 25.37% improvement in inference speed compared to processing entities one by one.
  • Qualitative Grounding: Visualization shows the model correctly identifying specific objects mentioned in complex sentences, such as distinguishing between a "yellow tennis suit" and "white tennis shoes."

Limitations and Future Work

  • Object Detector Dependency: The model's performance is upper-bounded by the quality of the underlying object detector (Faster R-CNN). If the detector misses an object, the attention mechanism cannot find it.
  • Small Objects & Complex Reasoning: The authors note that the model still struggles with very small objects (like a cigarette) or questions that require deep semantic reasoning rather than just visual grounding.
  • Future Directions: The authors suggest that BAN provides a new opportunity to learn richer joint representations for a wide variety of real-world multimodal problems beyond VQA and phrase localization.

Summary updated

Abstract

Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.
BibTeX
@inproceedings{kim2018bilinear,
  author = {Jin-Hwa Kim and Jaehyun Jun and Byoung-Tak Zhang},
  title = {{Bilinear Attention Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=HkW9NPbOZS},
}

Loading related content...

Figure