Multi-Level Contrastive Learning for Dense Prediction Task

Qiushan Guo, Yizhou Yu, Yi Jiang, Jiannan Wu, Zehuan Yuan, Ping Luo

Abstract

In this work, we present Multi-Level Contrastive Learning for Dense Prediction Task (MCL), an efficient self-supervised method for learning region-level feature representation for dense prediction tasks. Our method is motivated by the three key factors in detection: localization, scale consistency and recognition. To explicitly encode absolute position and scale information, we propose a novel pretext task that assembles multi-scale images in a montage manner to mimic multi-object scenarios. Unlike the existing image-level self-supervised methods, our method constructs a multi-level contrastive loss that considers each sub-region of the montage image as a singleton. Our method enables the neural network to learn regional semantic representations for translation and scale consistency while reducing pre-training epochs to the same as supervised pre-training. Extensive experiments demonstrate that MCL consistently outperforms the recent state-of-the-art methods on various datasets with significant margins. In particular, MCL obtains 42.5 AP$^{\mathrm{bb}}$ and 38.3 AP$^{\mathrm{mk}}$ on COCO with the 1x schedule fintuning, when using Mask R-CNN with R50-FPN backbone pre-trained with 100 epochs. In comparison to MoCo, our method surpasses their performance by 4.0 AP$^{\mathrm{bb}}$ and 3.1 AP$^{\mathrm{mk}}$. Furthermore, we explore the alignment between pretext task and downstream tasks. We extend our pretext task to supervised pre-training, which achieves a similar performance to self-supervised learning. This result demonstrates the importance of the alignment between pretext task and downstream tasks, indicating the potential for wider applicability of our method beyond self-supervised settings.

Introduction

Self-supervised learning (SSL) has revolutionized image-level classification, with methods like MoCo and SimCLR achieving performance comparable to supervised learning. However, these methods often struggle when applied to dense prediction tasks such as object detection and instance segmentation. The primary reason is a misalignment between the pretext task and the downstream requirements. Dense prediction requires precise localization, scale consistency (recognizing objects at varying sizes), and regional recognition.

Traditional SSL treats an entire image as a single entity, ignoring the spatial relationships and multi-object nature of detection scenes. While pixel-level or object-level SSL attempts have been made, they often suffer from high computational costs or reliance on low-quality object proposals. Furthermore, most state-of-the-art SSL methods require massive training budgets—often 800 to 1000 epochs—to become competitive, which is significantly longer than standard supervised pre-training.

Methodology: The MCL Framework

To bridge the gap between SSL and dense prediction, MCL introduces two primary innovations: Montage Assembly and Multi-Level Contrastive Learning.

Montage Assembly

Instead of processing a single augmented view of an image, MCL constructs a "montage." This involves taking multiple images, downsampling them to various scales, shuffling them, and packing them into a non-overlapping grid. This process mimics a real-world multi-object scenario and explicitly encodes absolute position and scale information into the training signal, forcing the model to learn spatial awareness.

MCL Framework: The architecture utilizes Montage Assembly and a Multi-Level Contrastive Loss across an FPN backbone.

Multi-Level Contrastive Learning

MCL utilizes a Feature Pyramid Network (FPN) backbone. Sub-regions of the montage are assigned to specific levels of the FPN based on their downsampling ratio. The model employs a contrastive loss (InfoNCE) to pull together features of the same sub-region across different scales and views while pushing away features from different sub-regions. A "Target Network," updated via Exponential Moving Average (EMA) of the "Online Network," ensures stable representation learning.

Experimental Results

The researchers evaluated MCL across a wide variety of benchmarks, including COCO, LVIS v1 (long-tail detection), Cityscapes, and ADE20K.

Efficiency and State-of-the-Art Performance

A standout feature of MCL is its efficiency. While traditional SSL methods require 800+ epochs to reach peak performance, MCL matches or exceeds supervised pre-training performance in just 100 epochs. On COCO, MCL achieves 42.5 $AP^{bb}$ and 38.3 $AP^{mk}$ with 100 epochs of pre-training on ImageNet, outperforming MoCo v2 by 4.0 AP.

Performance Comparison: MCL (red star) achieves higher mAP with significantly fewer pre-training epochs compared to other SSL methods.

Robustness and Generalization

  • Low-Data Regime: In scenarios with limited labels (1% of COCO), MCL achieves 12.1 AP, a significant jump over the 8.2 AP of standard supervised baselines.
  • Long-Tail Performance: On the LVIS dataset, MCL shows a +4.3 AP improvement on rare categories, indicating that its learned representations are highly generalizable and less biased toward frequent classes.
  • Architecture Agnostic: Although primarily tested on ResNet-50, the method also provides performance gains for Transformer-based architectures like Swin-T.

Conclusion

MCL demonstrates that the alignment between the pretext task and the downstream task is crucial for self-supervised success in dense prediction. By mimicking multi-object, multi-scale scenarios through Montage Assembly, MCL naturally learns the spatial and scale-invariant features required for detection and segmentation. Its ability to achieve state-of-the-art results in a fraction of the training time required by previous SSL methods makes it a highly practical framework for real-world computer vision applications. Future work may explore applying these multi-level principles to architectures beyond those using explicit Feature Pyramids.

Summary updated

Abstract

In this work, we present Multi-Level Contrastive Learning for Dense Prediction Task (MCL), an efficient self-supervised method for learning region-level feature representation for dense prediction tasks. Our method is motivated by the three key factors in detection: localization, scale consistency and recognition. To explicitly encode absolute position and scale information, we propose a novel pretext task that assembles multi-scale images in a montage manner to mimic multi-object scenarios. Unlike the existing image-level self-supervised methods, our method constructs a multi-level contrastive loss that considers each sub-region of the montage image as a singleton. Our method enables the neural network to learn regional semantic representations for translation and scale consistency while reducing pre-training epochs to the same as supervised pre-training. Extensive experiments demonstrate that MCL consistently outperforms the recent state-of-the-art methods on various datasets with significant margins. In particular, MCL obtains 42.5 AP$^\mathrm{bb}$ and 38.3 AP$^\mathrm{mk}$ on COCO with the 1x schedule fintuning, when using Mask R-CNN with R50-FPN backbone pre-trained with 100 epochs. In comparison to MoCo, our method surpasses their performance by 4.0 AP$^\mathrm{bb}$ and 3.1 AP$^\mathrm{mk}$. Furthermore, we explore the alignment between pretext task and downstream tasks. We extend our pretext task to supervised pre-training, which achieves a similar performance to self-supervised learning. This result demonstrates the importance of the alignment between pretext task and downstream tasks, indicating the potential for wider applicability of our method beyond self-supervised settings.
BibTeX
@misc{guo2023multi,
  author = {Qiushan Guo and Yizhou Yu and Yi Jiang and Jiannan Wu and Zehuan Yuan and Ping Luo},
  title = {{Multi-Level Contrastive Learning for Dense Prediction Task}},
  year = {2023},
  eprint = {2304.02010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.02010},
}
Figure