Good helper is around you: Attention-driven Masked Image Modeling

Zhengqi Liu, Jie Gui, Hao Luo

Abstract

Recently, masked image modeling (MIM), which learns visual representations by reconstructing the masked patches of an image, has dominated self-supervised learning in computer vision. However, the pre-training of MIM always takes massive time due to the large-scale data and large-size backbones. We mainly attribute it to the random patch masking in previous MIM works, which fails to leverage the crucial semantic information for effective visual representation learning. To tackle this issue, we propose the Frequency & Attention-driven Masking and Throwing Strategy (FAMT), which can extract semantic patches and reduce the number of training patches to boost model performance and training efficiency simultaneously. Specifically, FAMT utilizes the self-attention mechanism to extract semantic information from the image for masking during training in an unsupervised manner. However, attention alone could sometimes focus on inappropriate areas regarding the semantic information. Thus, we are motivated to incorporate the information from the frequency domain into the self-attention mechanism to derive the sampling weights for masking, which captures semantic patches for visual representation learning. Furthermore, we introduce a patch throwing strategy based on the derived sampling weights to reduce the training cost. FAMT can be seamlessly integrated as a plug-and-play module and surpasses previous works, e.g., reducing the training phase time by nearly 50% and improving the linear probing accuracy of MAE by 1.3% ~ 3.9% across various datasets, including CIFAR-10/100, Tiny ImageNet, and ImageNet-1K. FAMT also demonstrates superior performance in downstream detection and segmentation tasks.

Introduction

Masked Image Modeling (MIM), exemplified by frameworks like Masked Autoencoders (MAE), has revolutionized self-supervised learning in computer vision. By tasking a model with reconstructing hidden image patches, MIM forces the network to learn robust visual representations. However, standard MIM relies on random masking, which suffers from two primary inefficiencies:

  1. Semantic Redundancy: Random masking treats all patches equally, often forcing the model to expend significant computational resources on low-information regions like backgrounds (e.g., grass or sky) rather than the primary object.
  2. Training Cost: Due to the lack of semantic guidance, these models require extensive pre-training cycles and massive datasets to achieve high performance.

To address these challenges, we introduce FAMT (Frequency & Attention-driven Masking and Throwing), a strategy designed to intelligently identify and prioritize semantic patches while discarding redundant data.

Methodology: The FAMT Strategy

FAMT is a plug-and-play module that enhances existing MIM frameworks by replacing random masking with a semantic-aware approach. It consists of three core pillars:

1. Semantic Extraction via Attention

The strategy leverages self-attention maps from the encoder's final layer. Specifically, it analyzes the relationship between the [CLS] token and other patches to estimate the semantic importance of different image regions.

2. Frequency Domain Refinement

Because attention maps can be noisy or biased during early training, FAMT incorporates frequency-domain information. By applying a Fast Fourier Transform (FFT) and a low-pass filter, the model identifies low-frequency components that correlate strongly with global structure and semantic objects.

3. The Patch Throwing Strategy

Unlike standard MIM, which only masks or keeps patches, FAMT introduces a third category: Thrown patches.

  • Masked: High-importance patches are hidden to create a challenging reconstruction task.
  • Visible: Low-importance patches are kept to provide essential context.
  • Thrown: Medium-importance patches are discarded entirely. They are excluded from both the encoder and decoder, significantly reducing the computational footprint.
Comparison of standard MIM and the proposed FAMT strategy. FAMT introduces a "throwing" mechanism for medium-importance patches to reduce computational load.

Experimental Results

The effectiveness of FAMT was validated across multiple benchmarks, including ImageNet-1K, CIFAR-10/100, and downstream tasks like COCO (detection) and ADE20K (segmentation).

Visualization of semantic extraction. (a) Original image, (b) Attention map, (c) Frequency map, (d) Combined FAMT sampling weights.

Performance and Efficiency

  • Speedup: FAMT reduces training time by approximately 50% compared to standard MAE.
  • Accuracy Gains: On ImageNet-1K, FAMT improved linear probing accuracy by 1.3% to 3.9%.
  • Data Efficiency: The model achieved superior results using only 70% of the data volume required by the original MAE.
  • Downstream Tasks: In instance segmentation on the LVIS dataset, FAMT outperformed baseline MAE in both bounding box and mask AP metrics, demonstrating the robustness of its learned features.

Discussion and Conclusion

FAMT demonstrates that "harder" reconstruction tasks—specifically those targeting semantic regions—lead to better feature learning. By discarding redundant "middle-importance" patches, the model focuses its capacity on the most informative parts of the image. The integration of frequency-domain information acts as a stable prior, guiding the attention mechanism during the early stages of training.

While FAMT offers significant improvements, future work could explore the sensitivity of the masking and throwing ratios across specialized domains like medical imaging. Nonetheless, FAMT provides a highly efficient, scalable, and effective solution for modern self-supervised vision training.

Summary updated

Abstract

Recently, masked image modeling (MIM), which learns visual representations by reconstructing the masked patches of an image, has dominated self-supervised learning in computer vision. However, the pre-training of MIM always takes massive time due to the large-scale data and large-size backbones. We mainly attribute it to the random patch masking in previous MIM works, which fails to leverage the crucial semantic information for effective visual representation learning. To tackle this issue, we propose the Frequency \& Attention-driven Masking and Throwing Strategy (FAMT), which can extract semantic patches and reduce the number of training patches to boost model performance and training efficiency simultaneously. Specifically, FAMT utilizes the self-attention mechanism to extract semantic information from the image for masking during training in an unsupervised manner. However, attention alone could sometimes focus on inappropriate areas regarding the semantic information. Thus, we are motivated to incorporate the information from the frequency domain into the self-attention mechanism to derive the sampling weights for masking, which captures semantic patches for visual representation learning. Furthermore, we introduce a patch throwing strategy based on the derived sampling weights to reduce the training cost. FAMT can be seamlessly integrated as a plug-and-play module and surpasses previous works, \emph{e.g.} reducing the training phase time by nearly $50\%$ and improving the linear probing accuracy of MAE by $1.3\% \sim 3.9\%$ across various datasets, including CIFAR-10/100, Tiny ImageNet, and ImageNet-1K. FAMT also demonstrates superior performance in downstream detection and segmentation tasks.
BibTeX
@inproceedings{liu2023good,
  author = {Zhengqi Liu and Jie Gui and Hao Luo},
  title = {{Good helper is around you: Attention-driven Masked Image Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25269},
}
Figure