Bilinear Attention Networks

Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang

Introduction

Attention mechanisms in multimodal learning provide an efficient way to selectively utilize visual and textual information. However, calculating attention distributions for every possible pair of multimodal input channels is traditionally prohibitively expensive. To bypass this, previous approaches often relied on "co-attention," which builds separate attention distributions for each modality, neglecting the fine-grained interactions between them.

Bilinear Attention Networks (BAN) address this gap by seamlessly modeling the interactions between every possible pair of visual regions and textual words. By leveraging low-rank bilinear pooling and a novel multimodal residual learning skip-connection, BAN achieves state-of-the-art performance on major benchmarks like VQA 2.0 and Flickr30k Entities while remaining computationally efficient.

The Problem with Standard Co-Attention

In tasks like Visual Question Answering (VQA), a model must understand how specific words in a question relate to specific regions in an image. The problem with standard co-attention is that it often "squashes" information too early. It might identify "important words" and "important regions" independently, but it misses the nuanced links between them.

For example, if a question asks about a "red car" and a "blue truck," a standard model might attend to both vehicles and both colors but struggle to link "red" specifically to "car." Modeling every possible interaction—bilinear attention—is the ideal solution, but historically, it has been too computationally expensive and parameter-heavy to implement effectively.

Core Methodology

To make dense multimodal interaction practical, the authors introduce two primary innovations:

Architecture of Bilinear Attention Networks, illustrating Bilinear Attention Maps and Multimodal Residual Networks.
  1. Bilinear Attention Maps: Instead of a single attention vector, BAN computes a bilinear attention matrix that considers the relationship between all $k$ image regions and all $m$ words. To prevent the computational complexity from exploding, the model utilizes low-rank bilinear pooling. This technique factorizes large weight matrices into smaller, manageable components, allowing the network to learn complex joint representations without a massive increase in parameters.
  2. Multimodal Residual Networks (MRN): To improve the model's "reasoning" depth, the authors stack multiple "glimpses" (attention steps). Unlike previous works that simply concatenated these glimpses, BAN uses a residual learning approach. Each new attention map adds a refined representation to the previous one, similar to how ResNets operate in computer vision. This allows the model to scale up to 8 or even 12 glimpses effectively without suffering from optimization issues.

Key Insights and Performance

The core insight of BAN is Interaction over Compression: maintaining the multi-channel nature of both inputs (the full sequence of words and the full set of image objects) for as long as possible leads to better reasoning than compressing them into single vectors. Furthermore, by processing multi-channel inputs simultaneously through matrix operations rather than looping through entities, BAN achieved a 25.37% improvement in inference speed compared to previous state-of-the-art methods.

Validation scores demonstrating the performance of BAN across different epochs, parameter counts, and number of glimpses.

The model was evaluated on two major benchmarks, yielding impressive results:

  • VQA 2.0: Designed to reduce "language bias," this massive dataset requires genuine visual reasoning. BAN achieved a state-of-the-art score of 70.04% (when combined with additional features like GloVe embeddings and a counting module), significantly outperforming the 2017 challenge winner.
  • Flickr30k Entities: Used to evaluate "visual grounding" (localizing specific phrases to bounding boxes). The model set a new record for phrase localization with 69.69% Recall@1, a substantial 4.48% jump over the previous best method.

Qualitative visualizations also demonstrate the model's robustness, showing it successfully grounding complex queries—such as correctly identifying the specific "pants" belonging to a "guy skateboarding" among multiple people in a scene.

Limitations and Future Directions

Despite its success, BAN has a few limitations. The authors noted that the model still struggles with very small objects (e.g., a cigarette in a person's hand) because the underlying object detector might miss them or the attention map might be too coarse. Additionally, while BAN excels at physical grounding, it faces challenges with high-level semantic inference that requires broader contextual knowledge (e.g., identifying a "male conductor").

Looking forward, the bilinear attention framework holds significant promise. The authors suggest it could be extended to other "many-to-many" multimodal tasks beyond just vision and language, such as video processing or medical imaging, where dense interactions between different data streams are critical.

Summary updated

Abstract

Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.
BibTeX
@inproceedings{kim2018bilinear,
  author = {Jin-Hwa Kim and Jaehyun Jun and Byoung-Tak Zhang},
  title = {{Bilinear Attention Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=HkW9NPbOZS},
}
Figure