Contrastive Masked Autoencoders are Stronger Vision Learners

Zhicheng Huang, Xiaojie Jin, Chengze Lu, Qibin Hou, Ming-Ming Cheng, Dongmei Fu, Xiaohui Shen, Jiashi Feng

TL;DR

Contrastive Masked Autoencoders (CMAE) unify two powerhouse self-supervised learning paradigms—Masked Image Modeling (MIM) and Contrastive Learning (CL)—into a single, high-performance framework. By introducing a novel feature decoder and "pixel shifting" augmentation, CMAE produces vision representations that are both locally perceptive and globally discriminative, setting new state-of-the-art records on ImageNet, ADE20k, and COCO.

What Problem Does This Paper Address?

In the world of self-supervised vision learning, two methods currently dominate:

  1. Masked Image Modeling (MIM): Like a visual version of fill-in-the-blanks, models like MAE learn by reconstructing missing patches of an image. This makes them excellent at understanding local structures and spatial relations. However, they often lack "discriminability"—the ability to clearly distinguish one unique image instance from another.
  2. Contrastive Learning (CL): Methods like MoCo or SimCLR learn by comparing different views of the same image (positive pairs) and pushing them away from different images (negative pairs). This creates highly discriminative features but often ignores fine-grained local details.

Researchers have tried to combine these two, but simple "multi-task" approaches usually fail to deliver significant gains. The conflict lies in their requirements: CL thrives on heavy data augmentation (like aggressive cropping), while MIM needs stable, spatially consistent inputs to perform reconstruction. CMAE aims to resolve this conflict and get the best of both worlds.

Methods

CMAE uses a siamese architecture consisting of two main branches:

  • The Online Branch: An asymmetric encoder-decoder. The encoder receives a masked version of the image (only a few visible patches). It has two decoders:
  • Pixel Decoder: Reconstructs the raw pixels of the masked patches (the standard MIM task).
  • Feature Decoder: This is a key innovation. It predicts the latent features of the masked patches to "complete" the representation before it is used for contrastive learning.
  • The Momentum Branch: A momentum-updated version of the encoder that sees the full, unmasked image. This provides a stable, semantically complete target for the online branch to align with.
Fig. 3: Overall pipeline. Our method contains three components: the online encoder, momentum encoder, and online decoder. Given a training image, it applies pixel shifting to generate different views, which are then fed into the online and momentum encoders respectively. The online encoder randomly masks a fraction of the image patches and operates on the visible ones. The momentum encoder operates on the whole view after pixel shifting. The pixel decoder learns to reconstruct the input image from the image tokens (along with MASK tokens) provided by the online encoder, while the feature decoder learns to predict the features of the input image for contrastive learning with the momentum encoder output features. During pre-training, the parameters of the momentum encoder and projection head are updated using an exponential moving average algorithm. After the pre-training, only the online encoder is kept for downstream applications.

To make CL and MIM compatible, the authors introduced Pixel Shifting. Instead of the aggressive random cropping used in traditional CL—which can create views so different they confuse the MIM task—pixel shifting creates two views by slightly shifting the cropping window over a "master" image. This provides enough variance for contrastive learning while maintaining the spatial alignment necessary for reconstruction.

Datasets

The models were pre-trained on the massive ImageNet-1K dataset. To prove the versatility and transferability of the learned features, the authors evaluated CMAE on:

  • ImageNet-1K: For image classification.
  • ADE20k: For semantic segmentation.
  • COCO: For object detection and instance segmentation.
  • iNaturalist (2017/2018/2019) and Places365: For transfer learning across different domains.

Key Insights and Ideas

  • The Feature Decoder is the Missing Link: The authors found that you cannot effectively contrast the features of a partially masked image with a full image directly. The distribution gap is too wide. The feature decoder acts as a bridge, "filling in" the latent blanks so the contrastive task compares like-with-like.
  • Weak Augmentation is a Strength: In a surprising twist for contrastive learning, "weaker" augmentation (pixel shifting) performed better than heavy cropping. This is because it preserves the semantic integrity required for the MIM reconstruction task to remain meaningful.
  • Faster Convergence: CMAE isn't just more accurate; it's more efficient. During fine-tuning, CMAE reaches the final accuracy of a standard MAE model in roughly one-third of the time.

Key Results

CMAE consistently outperformed its predecessors across the board:

  • ImageNet-1K: CMAE-Base achieved 85.3% top-1 accuracy, surpassing the previous best (ConvMAE) by 0.7% and vanilla MAE by 1.1%.
  • ADE20k: It reached 52.5% mIoU, a substantial 1.8% improvement over prior state-of-the-art results.
  • COCO: It showed clear gains in both object detection (52.9 APb) and instance segmentation (47.0 APm).
Fig. 2: Comparisons with previous state-of-the-art MIM methods on ImageNet-1K in terms of top-1 accuracy at different pre- training epochs.

As shown in the figure above, CMAE maintains a significant performance lead over other methods like MAE, CAE, and iBOT across various pre-training durations.

Limitations and Future Work

While CMAE is highly effective, the authors note a few areas for growth:

  • Scaling: The current study focuses on standard model sizes (ViT-Base). Future work will explore how CMAE scales to massive datasets and "Huge" model architectures.
  • Multimodal Integration: The authors suggest incorporating other data types, such as image-dense captions, as additional "views" to further enrich the contrastive learning process.
  • Complexity: The architecture is more complex than a standard autoencoder, requiring a momentum branch and multiple decoders, which increases the computational footprint during the pre-training phase.

Summary updated

Abstract

Masked image modeling (MIM) has achieved promising results on various vision tasks. However, the limited discriminability of learned representation manifests there is still plenty to go for making a stronger vision learner. Towards this goal, we propose Contrastive Masked Autoencoders (CMAE), a new self-supervised pre-training method for learning more comprehensive and capable vision representations. By elaboratively unifying contrastive learning (CL) and masked image model (MIM) through novel designs, CMAE leverages their respective advantages and learns representations with both strong instance discriminability and local perceptibility. Specifically, CMAE consists of two branches where the online branch is an asymmetric encoder-decoder and the momentum branch is a momentum updated encoder. During training, the online encoder reconstructs original images from latent representations of masked images to learn holistic features. The momentum encoder, fed with the full images, enhances the feature discriminability via contrastive learning with its online counterpart. To make CL compatible with MIM, CMAE introduces two new components, i.e. pixel shifting for generating plausible positive views and feature decoder for complementing features of contrastive pairs. Thanks to these novel designs, CMAE effectively improves the representation quality and transfer performance over its MIM counterpart. CMAE achieves the state-of-the-art performance on highly competitive benchmarks of image classification, semantic segmentation and object detection. Notably, CMAE-Base achieves $85.3\%$ top-1 accuracy on ImageNet and $52.5\%$ mIoU on ADE20k, surpassing previous best results by $0.7\%$ and $1.8\%$ respectively. The source code is publicly accessible at \url{https://github.com/ZhichengHuang/CMAE}.
BibTeX
@article{huang2022contrastive,
  author = {Zhicheng Huang and Xiaojie Jin and Chengze Lu and Qibin Hou and Ming-Ming Cheng and Dongmei Fu and Xiaohui Shen and Jiashi Feng},
  title = {{Contrastive Masked Autoencoders are Stronger Vision Learners}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=I53gr3nDAx},
}
Figure