Area Attention

Yang Li, Lukasz Kaiser, Samy Bengio, Si Si

Area attention is a novel mechanism that allows deep learning models to attend to structurally adjacent groups of items—like phrases in a sentence or regions in an image—rather than just individual tokens or pixels. By dynamically learning the optimal granularity of attention, this often parameter-free approach consistently outperforms standard attention across machine translation and image captioning tasks.

The Problem with Item-Based Attention

Standard attention mechanisms are inherently "item-based." In a sentence, the model looks at individual words; in an image, it looks at individual grid cells. However, information is rarely contained in a single isolated point. A visual concept like "a colorful bird" spans many pixels, and a complex meaning in a sentence might span a multi-word phrase.

Forcing a model to attend to these items one by one is like trying to understand a painting by looking through a straw. While "multi-head attention" helps by looking at several points at once, it doesn't explicitly capture the structural adjacency (the "area") of those points. To bridge this gap, researchers propose Area Attention, allowing models to focus on coherent regions of varying shapes and sizes.

How Area Attention Works

The core idea is to redefine the "memory" the model looks at. Instead of a list of individual items, the memory becomes a collection of areas.

  1. Defining Areas:
  2. In 1D (text), an area is a contiguous span of tokens.
  3. In 2D (images), an area is any rectangular region of grid cells.
2D Area Attention: Grouping adjacent grid cells into rectangular regions.
  1. Keys and Values: To represent an area, the model calculates a "key" (the mean of all item keys in that area) and a "value" (the sum of all item values). In its basic form, this requires zero additional parameters, making it a "drop-in" improvement for existing architectures like Transformers or LSTMs.
  2. Efficiency Trick: Computing every possible rectangular area sounds computationally expensive ($O(N^2)$). To keep it fast, the authors use Summed Area Tables (also known as integral images). This classic computer vision technique allows the model to calculate the sum or mean of any rectangular area in constant time, keeping the overall complexity manageable.
  3. Rich Features: Beyond the basic version, the authors also experimented with adding "shape features," such as the area's height, width, and the standard deviation of its internal keys, to give the model more context about what it is observing.

Key Insights and Results

The authors evaluated Area Attention on Neural Machine Translation (NMT) using WMT’14 English-German (EN-DE) and English-French (EN-FR) datasets, as well as Image Captioning using MS COCO and Flickr 1K.

  • Granularity Matters: The model doesn't just look at bigger chunks; it learns when to look big and when to look small. This is particularly powerful for character-level translation, where grouping characters into "areas" effectively allows the model to discover sub-word structures on its own. On EN-FR character-level translation, Area Attention improved the BLEU score from 33.82 to 35.16.
  • Small Models, Big Gains: Area Attention provides a more significant boost to smaller, capacity-constrained models. It seems to act as a structural prior that helps the model learn more efficiently.
  • Structural Coherence: By forcing attention to be contiguous, the model is less likely to pick up on "noise" from unrelated parts of the input. Visualizations show that the model naturally gravitates toward semantically meaningful regions.
Qualitative results showing the model attending to a coherent "area" (the frisbee) to generate a caption.

In standard NMT tasks, Area Attention consistently beat regular attention baselines. For example, a "Transformer Big" model reached 29.77 BLEU on EN-DE, a significant jump over the state-of-the-art at the time. For image captioning, the models achieved higher CIDEr and ROUGE-L scores than strong benchmarks. Notably, the parameter-free version performed nearly as well as the version with extra learned features, proving the power of the underlying structural concept.

Limitations and Future Work

While highly effective, Area Attention does have a few limitations:

  • Hyperparameters: The maximum area size ($S$ for 1D, $H \times W$ for 2D) is a fixed hyperparameter that needs to be tuned for different tasks.
  • Computational Overhead: Even with Summed Area Tables making it efficient, Area Attention is still computationally heavier than standard attention because the number of "keys" the model must consider grows with the maximum area size.
  • Pooling Methods: The authors primarily used mean and sum pooling. While they briefly mention max-pooling, it is harder to implement efficiently with the Summed Area Table trick and remains an area for future exploration.

Summary updated

Abstract

Existing attention mechanisms are trained to attend to individual items in a collection (the memory) with a predefined, fixed granularity, e.g., a word token or an image grid. We propose area atten...
BibTeX
@inproceedings{li2019area,
  author = {Yang Li and Lukasz Kaiser and Samy Bengio and Si Si},
  title = {{Area Attention}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
}
Figure