Maximizing the Position Embedding for Vision Transformers with Global Average Pooling

Wonjun Lee, Bumsub Ham, Suhyun Kim

Abstract

Vision Transformers (ViTs) traditionally rely on a class token for classification, but recent trends favor Global Average Pooling (GAP) for better translation invariance. Simultaneously, layer-wise Position Embedding (PE) delivery has been proposed to enhance model expressiveness. Curiously, combining GAP with layer-wise PE results in a performance degradation. This paper investigates this conflict and uncovers a "counterbalancing" effect where PE serves as a numerical stabilizer for token embeddings. We introduce MPVG (Maximizing the Position Embedding with GAP), a method that injects PE into the final layer normalization. MPVG restores the counterbalancing mechanism in GAP-based models, leading to significant accuracy gains across classification, detection, and segmentation benchmarks.

The Performance Conflict in Vision Transformers

In the evolution of Vision Transformers, two architectural choices have gained prominence: Global Average Pooling (GAP) and Layer-wise Position Embeddings. GAP replaces the traditional [CLS] token by averaging all patch embeddings, which often improves the model's ability to handle translated objects. Layer-wise PE methods, such as LaPE, deliver position information to every transformer block rather than just the input layer, aiming for higher expressiveness.

However, researchers observed a counter-intuitive "conflicting result." While GAP improves the original ViT and Layer-wise PE improves the original ViT, using them together leads to a performance drop compared to using Layer-wise PE with a class token. Specifically, for a DeiT-Ti model, moving from a class token to GAP provides a +0.26% boost in the original setup, but results in a -0.16% drop when applied to a layer-wise architecture.

Performance conflict between GAP and Layer-wise PE, and the improvement achieved by MPVG.

The Discovery: PE as a Counterbalance

To understand this conflict, the authors performed a detailed heatmap analysis of token and position embeddings. They discovered that as data progresses through the transformer layers, token embeddings and position embeddings become increasingly negatively correlated. Correlation coefficients can reach as high as -0.95 in the final layers.

This suggests that Position Embeddings do more than just encode spatial coordinates; they act as a numerical "counterbalance." The PE effectively "pushes back" against high-magnitude dimensions in the token embeddings, helping to stabilize the data distribution before it reaches the final prediction head.

In models using a class token, this token preserves the directional alignment needed for this counterbalancing. However, in GAP-based models, the final averaging operation destroys this specific alignment. Without a final injection of PE, the Layer Normalization (LN) is forced to use its internal learnable parameters to compensate, which is less effective and leads to a loss of discriminative features.

The MPVG Method: Restoring Balance

The proposed solution, MPVG (Maximizing the Position Embedding with GAP), introduces a simple yet effective structural modification. Like previous layer-wise methods, it delivers PE to each block through independent Layer Normalizations. The key innovation is the "Last LN" fix: the raw, initial position embedding ($pos_0$) is delivered directly to the final Layer Normalization step, immediately preceding the Global Average Pooling and the prediction head.

Architectural comparison between standard ViT, LaPE, PVG, and the proposed MPVG.

By injecting PE at the very end, MPVG restores the counterbalancing mechanism that GAP would otherwise disrupt. This allows the model to retain spatial awareness and numerical stability right up to the final classification step.

Experimental Evaluation

The authors evaluated MPVG across a wide range of architectures and tasks to demonstrate its generalizability.

Image Classification

On ImageNet-1K, MPVG consistently outperformed baselines. For instance, DeiT-Ti saw an improvement from 72.14% to 73.51% (+1.37%). Similar gains were observed for DeiT-S (79.81% to 80.61%) and other architectures like CeiT and T2T-ViT.

Downstream Tasks

The benefits of MPVG extend beyond classification:

  • Object Detection (COCO): Using the ViT-Adapter-Ti framework, MPVG achieved a +0.6 increase in Box AP.
  • Semantic Segmentation (ADE20K): Within the UperNet framework, the method yielded a +1.14 increase in mIoU.

Conclusion

MPVG addresses a fundamental conflict in modern Vision Transformer design. By recognizing that Position Embeddings serve as a vital numerical stabilizer, the authors were able to reconcile Global Average Pooling with layer-wise architectures. The resulting method is simple to implement, requires negligible additional parameters, and provides consistent performance boosts across various computer vision benchmarks. While currently optimized for GAP-based models, the "counterbalancing" theory opens new avenues for understanding and optimizing embedding interactions in Transformer architectures.

Summary updated

BibTeX
@inproceedings{lee2025maximizing,
  author = {Wonjun Lee and Bumsub Ham and Suhyun Kim},
  title = {{Maximizing the Position Embedding for Vision Transformers with Global Average Pooling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i17.33997},
}
Figure