Introduction
In recent years, the computer vision landscape has fragmented. Convolutional Neural Networks (CNNs) were long the undisputed standard. Then, Vision Transformers (ViTs) arrived, using self-attention to capture long-range global context but requiring massive amounts of data and compute. Most recently, MLP-Mixers showed that even simple Multi-Layer Perceptrons could achieve competitive results.
On the surface, these three architectures seem completely disparate. Furthermore, pure Transformers struggle to scale to downstream tasks like object detection or high-resolution segmentation because their self-attention mechanism scales quadratically with image resolution.
This paper unifies CNNs, Vision Transformers, and MLP-Mixers under a single mathematical framework based on how they aggregate spatial context. By combining the static, local properties of CNNs with the dynamic, global properties of Transformers, the authors introduce the CONTAINER architecture, which achieves state-of-the-art image classification, faster convergence, and scales efficiently to high-resolution downstream tasks.
A Unified View of Context Aggregation
The authors demonstrate that CNNs, Transformers, and MLPs are actually special cases of a generalized contextual aggregation module. The core difference between them lies entirely in how they define their affinity matrix—the matrix that determines how information propagates between a query pixel and its surrounding context:
- Transformers use a dynamic affinity matrix generated on the fly by multiplying queries and keys.
- Depthwise CNNs use a static, sparse affinity matrix (local convolution kernels shared across the image).
- MLP-Mixers use a static, dense affinity matrix.
To leverage the strengths of all approaches, the authors propose the CONTAINER (CONText AggregatIon NEtwoRk) block. It computes a linear combination of a dynamic affinity matrix (like a Transformer) and a static affinity matrix (like a CNN/MLP), controlled by learnable parameters.
Because the full CONTAINER block is computationally heavy at high resolutions, they also introduce CONTAINER-LIGHT. This efficient variant uses a hierarchical structure that gradually downsamples the image. It manually switches off the expensive dynamic affinity matrix in the first three high-resolution stages, relying solely on static affinity until the final, lowest-resolution stage.
Key Insights: The Emergence of Locality
A powerful conceptual simplification of this work is realizing that CNNs, ViTs, and MLPs are just different flavors of affinity matrices. This allows researchers to mix and match these properties mathematically rather than treating them as competing dogmas.
Fascinatingly, when the CONTAINER network is allowed to learn a static affinity matrix from scratch, the early layers naturally evolve to look exactly like local convolutions. As shown above, the network independently "discovers" that paying attention to immediate neighboring pixels is the best way to process early-stage visual features. Static affinity provides the inductive bias needed for fast convergence and data efficiency (CNN traits), while dynamic affinity provides the capacity for complex, long-range global reasoning (Transformer traits).
Key Results
The CONTAINER architecture was evaluated on ImageNet-1k for classification and COCO for downstream object detection and instance segmentation.
- ImageNet Classification & Convergence: The base CONTAINER model achieves 82.7% Top-1 accuracy with 22M parameters, outperforming DeiT-Small by 2.8% with a comparable parameter count. Furthermore, CONTAINER hits DeiT-Small's peak accuracy (79.9%) in just 200 epochs instead of 300.
- Extreme Data Efficiency: When trained on just 10% of the ImageNet dataset, CONTAINER-LIGHT achieves 61.8% accuracy, absolutely crushing DeiT's 39.3% under the same constraints.
- Downstream Tasks (COCO): Because CONTAINER-LIGHT scales linearly at high resolutions, it serves as an excellent backbone. Plugged into RetinaNet, it yields 43.8 mAP (a massive +7.0 improvement over a comparable ResNet-50). Mask R-CNN and DETR show similarly huge leaps (+6.9 and +6.6 mAP, respectively).
Limitations and Future Work
The authors acknowledge that the full CONTAINER model cannot be directly applied to high-resolution inputs because the dynamic affinity matrix (self-attention) scales quadratically, requiring too much memory. To solve this, CONTAINER-LIGHT relies on a hand-crafted architectural choice—manually turning off dynamic attention in the first three stages.
For future work, the authors suggest developing methods where the network can learn when and where to switch off the dynamic affinity matrix based on the specific task at hand, rather than relying on hard-coded architectural stages.
Summary updated