Abstract
While the research community has focused extensively on the intricacies of self-attention mechanisms, the final aggregation step—pooling—is often treated as a minor architectural detail. This paper introduces a rigorous theoretical framework to quantify how different pooling strategies, such as Average, Max, and CLS, impact a Transformer model's expressivity. By deriving closed-form mathematical bounds based on Lipschitz continuity, we demonstrate that pooling acts as a critical "zoom level" for the model. Our findings, supported by extensive empirical evaluation across Computer Vision, Natural Language Processing, and Time Series analysis, reveal that the optimal pooling method is dictated by whether a task requires fine-grained local details or broad global context.
1. Introduction
Transformer models process inputs as sequences of tokens, generating a corresponding sequence of contextualized embeddings. However, most downstream applications—ranging from sentiment analysis to image classification—require these token-level embeddings to be aggregated into a single, fixed-size representation. This process, known as pooling, is a fundamental component of the model architecture.
Despite its ubiquity, pooling is frequently implemented using default settings (such as the [CLS] token or Average pooling) without a systematic understanding of its impact. This paper addresses this oversight by providing the first end-to-end theoretical analysis of how standard pooling mechanisms influence a Transformer's ability to distinguish between different inputs, bridging the gap between architectural choice and downstream performance.
2. Theoretical Framework: Quantifying Expressivity
The authors define "expressivity" as the model's capacity to map semantically similar inputs to nearby points in the output space while maintaining distance between dissimilar inputs. To quantify this, the study utilizes principles of Lipschitz continuity to derive mathematical bounds for standard pooling operations.
2.1 The "Zoom Level" of Pooling
The theoretical analysis reveals that the choice of pooling function acts as a leading multiplicative factor on the model's overall expressivity. This leads to a categorization of pooling methods based on their scaling properties:
- Contractive Pooling (Average): Scaling by $1/\sqrt{n}$, Average pooling smooths out individual token variations. This makes the representation highly stable and ideal for capturing broad, global structures.
- Expansive Pooling (Sum and Max): Scaling by $\sqrt{n}$ or $\sqrt{\min(n,d)}$, these methods amplify token-level variations. While this increases sensitivity to localized details, it also makes the model more susceptible to noise and perturbations.
- Non-expansive Pooling (Last-token/CLS): Scaling by a factor of 1, these methods preserve variations exactly as they appear in the final layer, acting as a neutral baseline for information preservation.
3. Empirical Validation
To validate these theoretical bounds, the researchers conducted experiments using frozen backbones from various state-of-the-art pre-trained Transformers, including ViT (Vision), GPT-2, Llama 3, Mistral (NLP), and MOMENT (Time Series). By attaching different pooling heads to these frozen models, the study isolated the effect of the pooling mechanism itself.
3.1 Multi-Modal Evaluation
The strategies were tested across three major domains:
- Natural Language Processing: Evaluated on tasks like the STS Benchmark, Banking77, and TinyStories.
- Computer Vision: Tested on ImageNet-100, CUB-200-2011, and CelebA (inpainting).
- Time Series: Analyzed using ETT datasets and various UCR archive classification tasks.
4. Key Results and Insights
The empirical data strongly aligned with the derived theoretical predictions, highlighting a clear trade-off between local sensitivity and global stability.
4.1 Task-Specific Performance
In NLP, global-context tasks such as the Banking77 classification dataset saw Average pooling dominate with 85.1% accuracy, significantly outperforming Last-token pooling (77.1%). Conversely, for local-context tasks like next-token prediction on TinyStories, Last-token pooling achieved 82.1% accuracy, while Average pooling dropped to 37.7%.
In Computer Vision, Average pooling proved superior for tasks requiring global structural understanding, such as image inpainting. However, CLS pooling remained the most effective for standard image classification. In the Time Series domain, Last-token pooling was the clear winner for forecasting—where the most recent data points are most critical—while Attention-based pooling excelled in classification by adaptively weighting specific segments of the sequence.
4.2 Learnable Pooling as a Chameleon
The study also explored learnable pooling methods, such as Weighted Average and Attention-based pooling. Interestingly, when provided with sufficient data, these methods naturally converged to mimic the optimal fixed pooling strategy for the given task. For example, in text classification, learnable weights became near-uniform (mimicking Average pooling), whereas in sequential tasks, they shifted weight to the final token.
5. Limitations and Future Directions
While this research provides a foundational understanding of pooling, the authors note that the experiments were conducted on frozen backbones. The interaction between pooling and the joint training of the entire Transformer architecture remains an area for further study. Additionally, while expressivity bounds are a necessary metric for model capacity, they do not inherently guarantee optimal generalization in every practical scenario.
Future work will focus on developing hybrid pooling methods that dynamically balance global smoothing with local sensitivity. Furthermore, the authors intend to investigate the relationship between expansive pooling and adversarial robustness, as theoretical models suggest that expansive methods may be more vulnerable to targeted noise.
Summary updated