Container: Context Aggregation Networks

Gao Peng, Jiasen Lu, Hongsheng Li, Roozbeh Mottaghi, Aniruddha KembhaviNeurIPS 2021 · OpenReview

Introduction

In recent years, the computer vision landscape has fragmented. Convolutional Neural Networks (CNNs) were long the undisputed standard. Then, Vision Transformers (ViTs) arrived, using self-attention to capture long-range global context but requiring massive amounts of data and compute. Most recently, MLP-Mixers showed that even simple Multi-Layer Perceptrons could achieve competitive results.

On the surface, these three architectures seem completely disparate. Furthermore, pure Transformers struggle to scale to downstream tasks like object detection or high-resolution segmentation because their self-attention mechanism scales quadratically with image resolution.

This paper unifies CNNs, Vision Transformers, and MLP-Mixers under a single mathematical framework based on how they aggregate spatial context. By combining the static, local properties of CNNs with the dynamic, global properties of Transformers, the authors introduce the CONTAINER architecture, which achieves state-of-the-art image classification, faster convergence, and scales efficiently to high-resolution downstream tasks.

A Unified View of Context Aggregation

The authors demonstrate that CNNs, Transformers, and MLPs are actually special cases of a generalized contextual aggregation module. The core difference between them lies entirely in how they define their affinity matrix—the matrix that determines how information propagates between a query pixel and its surrounding context:

  • Transformers use a dynamic affinity matrix generated on the fly by multiplying queries and keys.
  • Depthwise CNNs use a static, sparse affinity matrix (local convolution kernels shared across the image).
  • MLP-Mixers use a static, dense affinity matrix.

To leverage the strengths of all approaches, the authors propose the CONTAINER (CONText AggregatIon NEtwoRk) block. It computes a linear combination of a dynamic affinity matrix (like a Transformer) and a static affinity matrix (like a CNN/MLP), controlled by learnable parameters.

Because the full CONTAINER block is computationally heavy at high resolutions, they also introduce CONTAINER-LIGHT. This efficient variant uses a hierarchical structure that gradually downsamples the image. It manually switches off the expensive dynamic affinity matrix in the first three high-resolution stages, relying solely on static affinity until the final, lowest-resolution stage.

Key Insights: The Emergence of Locality

A powerful conceptual simplification of this work is realizing that CNNs, ViTs, and MLPs are just different flavors of affinity matrices. This allows researchers to mix and match these properties mathematically rather than treating them as competing dogmas.

Learned static affinity weights showing the emergence of locality

Fascinatingly, when the CONTAINER network is allowed to learn a static affinity matrix from scratch, the early layers naturally evolve to look exactly like local convolutions. As shown above, the network independently "discovers" that paying attention to immediate neighboring pixels is the best way to process early-stage visual features. Static affinity provides the inductive bias needed for fast convergence and data efficiency (CNN traits), while dynamic affinity provides the capacity for complex, long-range global reasoning (Transformer traits).

Key Results

The CONTAINER architecture was evaluated on ImageNet-1k for classification and COCO for downstream object detection and instance segmentation.

Training convergence comparing Container, Depthwise, and DeiT
  • ImageNet Classification & Convergence: The base CONTAINER model achieves 82.7% Top-1 accuracy with 22M parameters, outperforming DeiT-Small by 2.8% with a comparable parameter count. Furthermore, CONTAINER hits DeiT-Small's peak accuracy (79.9%) in just 200 epochs instead of 300.
  • Extreme Data Efficiency: When trained on just 10% of the ImageNet dataset, CONTAINER-LIGHT achieves 61.8% accuracy, absolutely crushing DeiT's 39.3% under the same constraints.
  • Downstream Tasks (COCO): Because CONTAINER-LIGHT scales linearly at high resolutions, it serves as an excellent backbone. Plugged into RetinaNet, it yields 43.8 mAP (a massive +7.0 improvement over a comparable ResNet-50). Mask R-CNN and DETR show similarly huge leaps (+6.9 and +6.6 mAP, respectively).

Limitations and Future Work

The authors acknowledge that the full CONTAINER model cannot be directly applied to high-resolution inputs because the dynamic affinity matrix (self-attention) scales quadratically, requiring too much memory. To solve this, CONTAINER-LIGHT relies on a hand-crafted architectural choice—manually turning off dynamic attention in the first three stages.

For future work, the authors suggest developing methods where the network can learn when and where to switch off the dynamic affinity matrix based on the specific task at hand, rather than relying on hard-coded architectural stages.

Summary updated

Abstract

Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations. Recently, Transformers -- originally introduced in natural language processing -- have been increasingly adopted in computer vision. While early adopters continued to employ CNN backbones, the latest networks are end-to-end CNN-free Transformer solutions. A recent surprising finding now shows that a simple MLP based solution without any traditional convolutional or Transformer components can produce effective visual representations. While CNNs, Transformers and MLP-Mixers may be considered as completely disparate architectures, we provide a unified view showing that they are in fact special cases of a more general method to aggregate spatial context in a neural network stack. We present the \model (CONText AggregatIon NEtwoRk), a general-purpose building block for multi-head context aggregation that can exploit long-range interactions \emph{a la} Transformers while still exploiting the inductive bias of the local convolution operation leading to faster convergence speeds, often seen in CNNs. Our \model architecture achieves 82.7 \% Top-1 accuracy on ImageNet using 22M parameters, +2.8 improvement compared with DeiT-Small, and can converge to 79.9 \% Top-1 accuracy in just 200 epochs. In contrast to Transformer-based methods that do not scale well to downstream tasks that rely on larger input image resolutions, our efficient network, named \modellight, can be employed in object detection and instance segmentation networks such as DETR, RetinaNet and Mask-RCNN to obtain an impressive detection mAP of 38.9, 43.8, 45.1 and mask mAP of 41.3, providing large improvements of 6.6, 7.3, 6.9 and 6.6 pts respectively, compared to a ResNet-50 backbone with a comparable compute and parameter size. Our method also achieves promising results on self-supervised learning compared to DeiT on the DINO framework. Code is released at https://github.com/allenai/container.
BibTeX
@inproceedings{peng2021container,
  author = {Gao Peng and Jiasen Lu and Hongsheng Li and Roozbeh Mottaghi and Aniruddha Kembhavi},
  title = {{Container: Context Aggregation Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=J9Rc5P4xjT},
}
Figure