Exploring the Coordination of Frequency and Attention in Masked Image Modeling

Jie Gui, Tuo Chen, Minjing Dong, Zhengqi Liu, Hao Luo, James Tin-Yau Kwok, Yuan Yan Tang

TL;DR

This paper introduces FAMT, a novel strategy for Masked Image Modeling (MIM) that significantly speeds up pre-training and improves performance by intelligently masking and "throwing away" less informative image patches. By combining self-attention with frequency domain analysis, FAMT focuses learning on semantically rich regions, leading to more efficient and effective visual representation learning.

What Problem Does This Paper Address?

Masked Image Modeling (MIM) has emerged as a powerful self-supervised learning technique for computer vision, enabling models to learn visual representations from unlabeled images by reconstructing masked portions. However, current MIM methods, like MAE and SimMIM, often rely on random masking, which can be inefficient. This random approach disperses the model's learning capacity across the entire image, including less informative background areas, leading to wasted computational resources and potentially weaker representations. Furthermore, the extensive pre-training required for these models on large datasets and large backbones makes them computationally expensive and time-consuming.

Methods

The core innovation of this paper is the Frequency & Attention-driven Masking and Throwing Strategy (FAMT). FAMT addresses the limitations of random masking by:

  • Semantic Information Extraction (Attention-driven Masking): It leverages the self-attention mechanism within Vision Transformers (ViTs) to generate attention maps. These maps highlight semantically important regions of the image, guiding the masking process to focus on these areas.
  • Frequency Domain Integration: Recognizing that attention alone might sometimes miss crucial information or overemphasize less important features, FAMT incorporates frequency domain analysis. By applying a low-pass filter and analyzing the low-frequency components of image tokens, it complements the attention-based weights. This combined approach ensures that both high-level semantic and low-level energy details are considered for masking.
  • Patch Throwing Strategy: To further reduce computational costs and improve training efficiency, FAMT introduces a "throwing" strategy. Based on the derived frequency and attention-based sampling weights, FAMT selectively discards (throws away) patches that are deemed less informative. This means these patches are not processed by the encoder or decoder, significantly reducing the computational load.

FAMT is designed as a plug-and-play module, meaning it can be seamlessly integrated into existing MIM frameworks like MAE and SimMIM without requiring substantial architectural changes.

Datasets

The paper uses a variety of standard datasets for training and evaluation:

  • Training: ImageNet-1K is primarily used for pre-training.
  • Evaluation (Linear Probing & Fine-tuning): CIFAR-10, CIFAR-100, Tiny ImageNet, ImageNet-1K, STL-10.
  • Downstream Tasks (Detection & Segmentation): COCO, LVIS, ADE20K, iSAID.

The paper does not introduce new benchmarks but rather demonstrates the effectiveness of FAMT on existing ones.

Key Insights and Ideas

  • Attention is not enough: While self-attention is powerful for identifying salient regions, it can sometimes be distracted by less relevant features or miss subtle but important details. Combining it with frequency domain information provides a more robust signal for masking.
  • Not all patches are created equal: The core idea is that some image patches are more critical for learning representations than others. Randomly masking all patches equally is inefficient. FAMT's approach of prioritizing semantically rich patches and discarding others is a more intelligent use of computational resources.
  • Throwing away data can be good: The "patch throwing" strategy is counter-intuitive but effective. By intelligently removing less informative patches, FAMT reduces the input size and computational burden, leading to faster training without sacrificing performance. This is particularly impactful for transformer architectures, which have quadratic complexity with respect to input sequence length.
  • Frequency domain complements attention: The paper highlights how ViTs have different behaviors in the frequency domain compared to CNNs. Leveraging this difference, specifically the low-frequency components, helps to create a more comprehensive masking strategy.

Key Results

FAMT demonstrates significant improvements across various aspects:

  • Training Efficiency: FAMT can reduce training phase time by nearly 50% (specifically, by using only 60% of the image data).
  • Linear Probing Accuracy: It improves the linear probing accuracy of MAE by 1.3% to 3.9% on datasets like CIFAR-10/100, Tiny ImageNet, and ImageNet-1K. In some ablation studies, improvements reach up to 5.9%.
  • Downstream Task Performance: FAMT shows superior performance in downstream detection and segmentation tasks on datasets like COCO, LVIS, ADE20K, and iSAID. For instance, it achieves a 1.1% improvement in mIOU on iSAID.
  • Ablation Studies:
    • The throwing strategy alone significantly improves linear probing accuracy.
  • Combining frequency and attention-driven masking (FAMT) generally outperforms attention-driven masking alone (FAM).
  • The default setting of discarding 40% of image patches (throwing ratio t=0.4) with a 45% masking ratio (r=0.45) yields strong results.

Limitations and Future Work

  • Small Object Segmentation: The paper notes that FAMT's throwing strategy might be less effective for segmenting very small objects, as it might discard informative parts. This is observed in the iSAID results where FAM (without throwing) showed better performance on small object segmentation.
  • Hyperparameter Sensitivity: While FAMT's approach offers a wider range of effective hyperparameters, finding the optimal settings still requires some experimentation.
  • Future Work: The authors suggest exploring the impact of throwing different areas of the image and further investigating the interplay between masking and throwing strategies. They also implicitly suggest that FAMT's plug-and-play nature makes it a valuable tool for future research in MIM.

Summary updated

Abstract

Recently, masked image modeling (MIM), which learns visual representations by reconstructing the masked patches of an image, has become a popular self-supervised paradigm. However, the pre-training of MIM always takes massive time due to the large-scale data and large-size backbones. We mainly attribute it to the random patch masking in previous MIM works, which fails to leverage the crucial semantic information for effective visual representation learning. To tackle this issue, we propose the Frequency & Attention-driven Masking and Throwing Strategy (FAMT), which can detect semantic patches and reduce the number of training patches to boost model performance and training efficiency simultaneously. Specifically, FAMT utilizes the self-attention mechanism to extract semantic information from the image for masking during training in an unsupervised manner. However, attention alone could sometimes focus on inappropriate areas regarding the semantic information. Thus, we are motivated to incorporate the information from the frequency domain into the self-attention mechanism to derive the sampling weights for masking, which captures semantic patches for visual representation learning. Furthermore, we introduce a patch throwing strategy based on the derived sampling weights to reduce the training cost. FAMT can be seamlessly integrated as a plug-and-play module and surpasses previous works, e.g. reducing the training phase time by nearly 50% and improving the linear probing accuracy of MAE by $1.8$ % ~ $ 6.3$ % across various datasets, including CIFAR-10/100, Tiny ImageNet, and ImageNet-1K. FAMT also demonstrates superior performance in downstream detection and segmentation tasks.
BibTeX
@inproceedings{gui2023exploring,
  author = {Jie Gui and Tuo Chen and Minjing Dong and Zhengqi Liu and Hao Luo and James Tin-Yau Kwok and Yuan Yan Tang},
  title = {{Exploring the Coordination of Frequency and Attention in Masked Image Modeling}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i2.25269},
}
Figure