Attention, Please! Revisiting Attentive Probing for Masked Image Modeling

Bill Psomas, Dionysis Christopoulos, Eirini Baltzi, Ioannis Kakogeorgiou, Tilemachos Aravanis, Nikos Komodakis, Konstantinos Karantzalos, Yannis Avrithis, Giorgos Tolias

Abstract

As fine-tuning (FT) becomes increasingly impractical at scale, probing is emerging as the preferred evaluation protocol for self-supervised learning (SSL). Yet, the standard linear probing (LP) fails to adequately reflect the potential of models trained with Masked Image Modeling (MIM), due to the distributed nature of patch tokens. This motivates the need for attentive probing, an alternative that uses attention to selectively aggregate patch-level features. Despite its growing adoption, attentive probing remains under-explored, with existing methods suffering from excessive parameterization and poor computational efficiency. In this work, we revisit attentive probing through the lens of the accuracy-efficiency trade-off. We conduct a systematic study of existing methods, analyzing their mechanisms and benchmarking their performance. We introduce efficient probing (EP), a multi-query cross-attention mechanism that eliminates redundant projections, reduces the number of trainable parameters, and achieves up to a 10$\times$ speed-up over conventional multi-head attention. Despite its simplicity, EP outperforms LP and prior attentive probing approaches across seven benchmarks, generalizes well beyond MIM to diverse pre-training paradigms, produces interpretable attention maps, and achieves strong gains in low-shot and layer-wise settings.

Introduction

The evaluation of large-scale vision models typically relies on Linear Probing (LP), where a classifier is trained on top of a frozen global representation (e.g., the [CLS] token). While efficient, LP assumes that all relevant information is compressed into a single vector. This assumption often fails for models trained via Masked Image Modeling (MIM), such as MAE, where knowledge is spatially distributed across patch tokens. Consequently, LP significantly underestimates the capabilities of such models.

Attentive probing offers a solution by using attention mechanisms to aggregate local features. However, current implementations like Multi-Head Cross-Attention (MHCA) are often computationally heavy and over-parameterized. This work introduces Efficient Probing (EP), a streamlined attention mechanism designed to maximize accuracy while minimizing computational overhead.

Methodology: Efficient Probing (EP)

Efficient Probing (EP) simplifies the standard attention mechanism by focusing on essential components. Unlike standard Multi-Head Cross-Attention (MHCA), which projects inputs into separate Query, Key, and Value spaces, EP utilizes a set of learnable query vectors ($u_j$) that exist in the same space as the input features.

Comparison of standard Multi-Head Cross-Attention (left) and the proposed Efficient Probing (right).

As shown in Figure 1, EP eliminates redundant projection matrices ($W_Q, W_K$). The learnable queries interact directly with frozen patch features to compute attention maps, which are then used to aggregate the value-projected features. This design drastically reduces the number of trainable parameters and FLOPs, achieving up to a 10$\times$ speed-up over conventional MHCA while maintaining high performance.

Experimental Results

The authors benchmarked EP across seven diverse datasets, including ImageNet-1K and fine-grained tasks like Food-101. The results demonstrate that EP consistently defines the state-of-the-art Pareto frontier for the accuracy-efficiency trade-off.

Accuracy vs. parameter count for various probing methods on ImageNet-1K (a) and Food-101 (b).

Key findings include:

  • Performance Gains: On ImageNet-1K with a MAE ViT-B backbone, EP achieves 75.6% top-1 accuracy, a massive improvement over the 67.7% achieved by standard LP.
  • Efficiency: As illustrated in Figure 4, EP variants (e.g., $D_o = D_i/4$) often outperform more complex methods like AIM/MHCA while using significantly fewer parameters.
  • Interpretability: EP's multiple queries learn to attend to complementary parts of objects (e.g., different parts of a bird), which also enables superior zero-shot object localization (+9.8% over baselines).
  • Generalization: EP proves effective across various pre-training paradigms, including DINO, CLIP, and generative models (DiT).

Conclusion

Efficient Probing (EP) addresses the limitations of standard linear probing by providing a lightweight, high-performance alternative for evaluating vision models. By rethinking the attention mechanism for probing, EP unlocks "hidden" knowledge in models like MAE and provides a more accurate reflection of their learned representations. Future work includes extending EP to dense prediction tasks like object detection and segmentation.

Summary updated

Abstract

As fine-tuning (FT) becomes increasingly impractical at scale, probing is emerging as the preferred evaluation protocol for self-supervised learning (SSL). Yet, the standard linear probing (LP) fails to adequately reflect the potential of models trained with Masked Image Modeling (MIM), due to the distributed nature of patch tokens. This motivates the need for attentive probing, an alternative that uses attention to selectively aggregate patch-level features. Despite its growing adoption, attentive probing remains under-explored, with existing methods suffering from excessive parameterization and poor computational efficiency. In this work, we revisit attentive probing through the lens of the accuracy-efficiency trade-off. We conduct a systematic study of existing methods, analyzing their mechanisms and benchmarking their performance. We introduce efficient probing (EP), a multi-query cross-attention mechanism that eliminates redundant projections, reduces the number of trainable parameters, and achieves up to a 10$\times$ speed-up over conventional multi-head attention. Despite its simplicity, EP outperforms LP and prior attentive probing approaches across seven benchmarks, generalizes well beyond MIM to diverse pre-training paradigms, produces interpretable attention maps, and achieves strong gains in low-shot and layer-wise settings. Code available at https://github.com/billpsomas/efficient-probing.
BibTeX
@inproceedings{psomas2025attention,
  author = {Bill Psomas and Dionysis Christopoulos and Eirini Baltzi and Ioannis Kakogeorgiou and Tilemachos Aravanis and Nikos Komodakis and Konstantinos Karantzalos and Yannis Avrithis and Giorgos Tolias},
  title = {{Attention, Please! Revisiting Attentive Probing for Masked Image Modeling}},
  booktitle = {Greeks in AI},
  year = {2025},
  url = {https://openreview.net/forum?id=I4S8KsFj20},
}

Loading related content...

Figure