Efficient Representation Learning via Adaptive Context Pooling

Chen Huang, Walter Talbott, Navdeep Jaitly, Josh M. Susskind

Abstract

Standard self-attention mechanisms in Transformers and convolutional layers in ConvNets operate on a fixed granularity, treating every input token or pixel with the same spatial or temporal scale. This paper introduces ContextPool, a lightweight, drop-in module designed to allow models to adaptively pool neighboring information for each token. By learning optimal context scales and weights for individual tokens, ContextPool enables models to achieve state-of-the-art performance with fewer layers and significantly lower computational overhead. Our experiments across ImageNet-1K, WMT translation tasks, and character-level language modeling demonstrate that ContextPool consistently improves accuracy and efficiency.

1. Introduction

In natural language processing and computer vision, the importance of a token is often defined by its surrounding context. However, standard architectures like the Transformer apply a uniform attention mechanism across all tokens, regardless of their specific contextual needs. For instance, a pronoun in a complex sentence may require a broad window of context to resolve its referent, whereas a specific noun might be self-contained. Similarly, in images, a patch representing a textured background requires a different "zoom" level than a patch containing a sharp object edge.

To compensate for this fixed granularity, researchers typically stack many layers to build up hierarchical representations. We propose that making tokens "context-aware" earlier in the pipeline via adaptive pooling can reduce the reliance on depth, leading to more efficient and performant models.

2. The ContextPool Module

ContextPool (CP) is positioned immediately before the attention layer. It transforms a sequence of raw tokens into a sequence of contextualized tokens through three primary mechanisms:

2.1 Adaptive Weights

Rather than using static average pooling, ContextPool learns a mapping function—implemented via two lightweight convolutional layers—to predict unique pooling weights for every token based on its immediate neighbors. This allows the model to prioritize specific nearby features that are most relevant to the current token.

2.2 Adaptive Support Size

The module predicts a "pooling size" for each token, which determines the standard deviation of a Gaussian mask. This mask acts as a soft, localized window, ensuring the model focuses on relevant local structures while filtering out distant noise. This "locality prior" is crucial for maintaining focus in high-dimensional feature spaces.

2.3 The "Pool then Attend" Workflow

For each token $x_i$, ContextPool aggregates features from its neighborhood using the learned weights and the Gaussian mask. The resulting contextual feature $y_i$ is then passed into the standard self-attention mechanism. Because ContextPool relies on simple convolutions, it adds negligible FLOPs compared to the quadratic cost of standard self-attention.

3. Experimental Evaluation

We evaluated ContextPool across several domains to verify its versatility and efficiency.

3.1 Image Classification

On the ImageNet-1K benchmark, a ViT-B/16 model equipped with ContextPool reached 79.9% Top-1 accuracy, a significant 2% improvement over the vanilla baseline (77.9%) with almost no additional computational cost. Furthermore, when integrated into ResNet architectures on CIFAR-10, ContextPool outperformed specialized variants like Deformable ConvNets.

3.2 Language Modeling and Translation

In character-level language modeling (enwik8), a 14-layer CP-Transformer achieved a state-of-the-art Bits Per Character (BPC) of 0.97, outperforming larger models such as Transformer-XL and Longformer. In Neural Machine Translation (WMT 2014 EN-DE and EN-FR), CP-models consistently delivered higher BLEU scores while maintaining faster inference speeds and lower memory footprints than standard attention or Area Attention baselines.

4. Key Insights

  • Efficiency through Granularity: By allowing the model to determine the necessary context for each token, we can reduce model depth. An 8-layer CP-Transformer matched the performance of a 10-layer vanilla Transformer while being 27% faster.
  • The Importance of Locality: The use of a Gaussian mask (locality prior) proved superior to random sparse patterns or no prior at all, suggesting that local context provides a vital inductive bias even in global attention models.
  • Emergent Hierarchy: Visual analysis reveals that learned pooling sizes generally increase in deeper layers. The model naturally learns a hierarchical structure—focusing on fine details in early layers and broader semantic concepts in later layers, mimicking biological visual systems.

5. Conclusion

ContextPool offers a simple yet powerful method for enhancing representation learning by addressing the bottleneck of fixed granularity. Its ability to be dropped into existing Transformer and ConvNet architectures makes it a highly practical tool for improving model performance and efficiency. Future research will explore its application in other high-frequency modalities, such as speech recognition, and its potential to further optimize hybrid architecture designs.

Summary updated

Abstract

Self-attention mechanisms model long-range context by using pairwise attention between all input tokens. In doing so, they assume a fixed attention granularity defined by the individual tokens (e.g., text characters or image pixels), which may not be optimal for modeling complex dependencies at higher levels. In this paper, we propose ContextPool to address this problem by adapting the attention granularity for each token. Inspired by the success of ConvNets that are combined with pooling to capture long-range dependencies, we learn to pool neighboring features for each token before computing attention in a given attention layer. The pooling weights and support size are adaptively determined, allowing the pooled features to encode meaningful context with varying scale. We show that ContextPool makes attention models more expressive, achieving strong performance often with fewer layers and thus significantly reduced cost. Experiments validate that our ContextPool module, when plugged into transformer models, matches or surpasses state-of-the-art performance using less compute on several language and image benchmarks, outperforms recent works with learned context sizes or sparse attention patterns, and is also applicable to ConvNets for efficient feature learning.
BibTeX
@inproceedings{huang2022efficient,
  author = {Chen Huang and Walter Talbott and Navdeep Jaitly and Josh M. Susskind},
  title = {{Efficient Representation Learning via Adaptive Context Pooling}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=vZLkx-6Hp_q},
}
Figure