Contextual Image Masking Modeling via Synergized Contrasting without View Augmentation for Faster and Better Visual Pretraining

Shaofeng Zhang, Feng Zhu, Rui Zhao, Junchi YanICLR 2023 · OpenReview

Introduction

ccMIM (contrasting-aided contextual Masked Image Modeling) is a novel self-supervised learning framework designed to achieve faster and more accurate visual pretraining by intentionally making the "fill-in-the-blanks" task harder. Instead of masking image patches at random, ccMIM uses a single-view contrastive loss to identify and mask the most semantically important parts of an image. This forces the model to reconstruct complex foreground objects from simple backgrounds, leading to richer feature representations.

The Problem: Bridging Two SSL Paradigms

In the realm of self-supervised learning (SSL) for computer vision, two dominant paradigms exist: Contrastive Learning and Masked Image Modeling (MIM).

  • Contrastive methods (e.g., SimCLR, MoCo) learn by comparing two different augmented "views" of the same image. They excel at capturing global context but often miss fine-grained local details and rely heavily on tedious, hand-crafted data augmentations.
  • MIM methods (e.g., MAE) learn by masking random patches of an image and training the model to reconstruct them. While excellent at learning local features, they converge very slowly—often requiring 1,600 or more epochs—and can struggle with global scene understanding.

Previous attempts to combine these approaches have typically just added their respective losses together without establishing a deeper connection. The authors of ccMIM argue for a synergistic approach: the contrastive task should help the masking task decide what to hide, while the masking task should make the reconstruction challenging enough to learn powerful features quickly.

Methods: A Synergistic Closed-Loop System

The core of ccMIM is a "closed-loop" architecture consisting of two main innovations: contextual masking and single-view token alignment.

Architecture of the ccMIM framework showing importance sampling and single-view token alignment.
  1. Contextual Masking via Importance Sampling: Unlike standard MIM models that mask patches randomly, ccMIM evaluates the attention scores from the model’s own [CLS] token to identify patches containing the most semantic information (e.g., the main subject of the image). It then uses importance sampling to preferentially mask these high-value patches. This significantly increases the difficulty of the reconstruction task, as the model must "hallucinate" the most important parts of the image based primarily on less-informative background context.
  2. Single-View Token Alignment: To guide the patch selection process and accelerate convergence, the authors introduce a contrastive loss that operates on a single view of the image. It aligns the global features (the [CLS] tokens) extracted from the masked patches with those from the visible patches. This acts as a regularizer, ensuring the model captures global information across both sets of patches without needing the complex data augmentations (like crops, flips, or color jitters) required by traditional contrastive learning.

By combining these mechanisms, the model operates in a continuous loop: the contrastive alignment provides semantic clues that guide the importance sampling, which in turn creates a harder reconstruction task that refines the model's feature representation.

Key Insights

  • "Harder is Better": Masking the background is often a wasted learning opportunity. If a model masks a bird but leaves the sky visible, it doesn't have to work hard to understand the bird's structure. Contextual masking forces the model to learn deep semantic representations.
  • Efficiency through Synergy: By using the contrastive loss to guide the masking process, the two objectives actively assist each other. This synergy allows the model to achieve higher accuracy in significantly fewer training cycles.
  • Augmentation-Free Contrasting: ccMIM avoids the pitfalls of traditional contrastive methods—which are highly sensitive to how images are distorted—by treating the "masked set" and "visible set" of a single image as the two views.

Key Results

The ccMIM framework was evaluated across several standard benchmarks, including ImageNet-1K, MS-COCO, and ADE20K, demonstrating state-of-the-art performance.

Performance comparison on ImageNet-1K classification, highlighting ccMIM's superior accuracy and linear probing results.
  • Superior Accuracy & Faster Convergence: On ImageNet-1K, ccMIM achieved 84.2% top-1 accuracy after 800 epochs of pretraining. Remarkably, ccMIM reached 83.6% accuracy in just 300 epochs, matching the performance of the baseline MAE model trained for 1,600 epochs—a roughly 5x speedup in training efficiency.
  • Stronger Global Features: In linear probing evaluations (where the backbone is frozen and only a final classifier is trained), ccMIM outperformed MAE by +4.5%, proving that its single-view token alignment successfully captures superior global representations.
  • Downstream Dominance: The model set new state-of-the-art results for MIM-based methods on dense vision tasks, including object detection on MS-COCO and semantic segmentation on ADE20K, proving its features are highly transferable.

Limitations and Future Work

While highly effective, the authors note a few limitations. First, importance-based masking can slightly slow down the very beginning of training because the initial reconstruction task is exceptionally difficult. This is mitigated by using "importance sampling" to retain some randomness rather than strictly masking the top-ranked patches. Additionally, the current architecture is heavily tailored to Vision Transformers (ViTs). Exploring how these contextual masking strategies might apply to convolutional neural networks (CNNs) or hybrid architectures remains an exciting avenue for future research.

Summary updated

Abstract

We propose a new contextual masking image modeling (MIM) approach called contrasting-aided contextual MIM (ccMIM), under the MIM paradigm for visual pretraining. Specifically, we adopt importance sampling to select the masked patches with richer semantic information for reconstruction, instead of random sampling as done in previous MIM works. As such, the resulting patch reconstruction task from the remaining less semantic patches could be more difficult and helps to learn. To speed up the possibly slowed convergence due to our more difficult reconstruction task, we further propose a new contrastive loss that aligns the tokens of the vision transformer extracted from the selected masked patches and the remaining ones, respectively. The hope is that it serves as a regularizer for patch feature learning such that the image-level global information could be captured in both masked and unmasked patches, and notably such a single-view contrasting avoids the tedious image augmentation step required in recent efforts of introducing contrastive learning to MIM (to speedup convergence and discriminative ability). Meanwhile, the attention score from the contrastive global feature can also carry effective semantic clues to in turn guide our above masking patch selection scheme. In consequence, our contextual MIM and contrastive learning are synergetically performed in a loop (semantic patch selection-token alignment contrasting) to boost the best of the two worlds: fast convergence and strong performance on downstream tasks without ad-hoc augmentations, which are verified by empirical results on ImageNet-1K for both classification and dense vision tasks.
BibTeX
@inproceedings{zhang2023contextual,
  author = {Shaofeng Zhang and Feng Zhu and Rui Zhao and Junchi Yan},
  title = {{Contextual Image Masking Modeling via Synergized Contrasting without View Augmentation for Faster and Better Visual Pretraining}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=A3sgyt4HWp},
}
Figure