Abstract
Standard self-attention mechanisms in Transformers and convolutional layers in ConvNets operate on a fixed granularity, treating every input token or pixel with the same spatial or temporal scale. This paper introduces ContextPool, a lightweight, drop-in module designed to allow models to adaptively pool neighboring information for each token. By learning optimal context scales and weights for individual tokens, ContextPool enables models to achieve state-of-the-art performance with fewer layers and significantly lower computational overhead. Our experiments across ImageNet-1K, WMT translation tasks, and character-level language modeling demonstrate that ContextPool consistently improves accuracy and efficiency.
1. Introduction
In natural language processing and computer vision, the importance of a token is often defined by its surrounding context. However, standard architectures like the Transformer apply a uniform attention mechanism across all tokens, regardless of their specific contextual needs. For instance, a pronoun in a complex sentence may require a broad window of context to resolve its referent, whereas a specific noun might be self-contained. Similarly, in images, a patch representing a textured background requires a different "zoom" level than a patch containing a sharp object edge.
To compensate for this fixed granularity, researchers typically stack many layers to build up hierarchical representations. We propose that making tokens "context-aware" earlier in the pipeline via adaptive pooling can reduce the reliance on depth, leading to more efficient and performant models.
2. The ContextPool Module
ContextPool (CP) is positioned immediately before the attention layer. It transforms a sequence of raw tokens into a sequence of contextualized tokens through three primary mechanisms:
2.1 Adaptive Weights
Rather than using static average pooling, ContextPool learns a mapping function—implemented via two lightweight convolutional layers—to predict unique pooling weights for every token based on its immediate neighbors. This allows the model to prioritize specific nearby features that are most relevant to the current token.
2.2 Adaptive Support Size
The module predicts a "pooling size" for each token, which determines the standard deviation of a Gaussian mask. This mask acts as a soft, localized window, ensuring the model focuses on relevant local structures while filtering out distant noise. This "locality prior" is crucial for maintaining focus in high-dimensional feature spaces.
2.3 The "Pool then Attend" Workflow
For each token $x_i$, ContextPool aggregates features from its neighborhood using the learned weights and the Gaussian mask. The resulting contextual feature $y_i$ is then passed into the standard self-attention mechanism. Because ContextPool relies on simple convolutions, it adds negligible FLOPs compared to the quadratic cost of standard self-attention.
3. Experimental Evaluation
We evaluated ContextPool across several domains to verify its versatility and efficiency.
3.1 Image Classification
On the ImageNet-1K benchmark, a ViT-B/16 model equipped with ContextPool reached 79.9% Top-1 accuracy, a significant 2% improvement over the vanilla baseline (77.9%) with almost no additional computational cost. Furthermore, when integrated into ResNet architectures on CIFAR-10, ContextPool outperformed specialized variants like Deformable ConvNets.
3.2 Language Modeling and Translation
In character-level language modeling (enwik8), a 14-layer CP-Transformer achieved a state-of-the-art Bits Per Character (BPC) of 0.97, outperforming larger models such as Transformer-XL and Longformer. In Neural Machine Translation (WMT 2014 EN-DE and EN-FR), CP-models consistently delivered higher BLEU scores while maintaining faster inference speeds and lower memory footprints than standard attention or Area Attention baselines.
4. Key Insights
- Efficiency through Granularity: By allowing the model to determine the necessary context for each token, we can reduce model depth. An 8-layer CP-Transformer matched the performance of a 10-layer vanilla Transformer while being 27% faster.
- The Importance of Locality: The use of a Gaussian mask (locality prior) proved superior to random sparse patterns or no prior at all, suggesting that local context provides a vital inductive bias even in global attention models.
- Emergent Hierarchy: Visual analysis reveals that learned pooling sizes generally increase in deeper layers. The model naturally learns a hierarchical structure—focusing on fine details in early layers and broader semantic concepts in later layers, mimicking biological visual systems.
5. Conclusion
ContextPool offers a simple yet powerful method for enhancing representation learning by addressing the bottleneck of fixed granularity. Its ability to be dropped into existing Transformer and ConvNet architectures makes it a highly practical tool for improving model performance and efficiency. Future research will explore its application in other high-frequency modalities, such as speech recognition, and its potential to further optimize hybrid architecture designs.
Summary updated