Multi-Level Contrastive Learning for Dense Prediction Task

Qiushan Guo, Yizhou Yu, Yi Jiang, Jiannan Wu, Zehuan Yuan, Ping LuoSubmission to ICLR 2024 · Withdrawn

TL;DR

Multi-Level Contrastive Learning (MCL) is an efficient self-supervised pre-training method designed specifically for dense prediction tasks like object detection and segmentation. By assembling multi-scale images into a non-overlapping grid "montage," MCL provides ground-truth localization and scale variation without requiring noisy bounding box proposals, matching or exceeding state-of-the-art transfer performance in just 100 ImageNet epochs.

What Problem Does This Paper Address?

Standard self-supervised learning (SSL) techniques—such as MoCo, SimCLR, and BYOL—focus on image-level instance discrimination. While they excel at image classification, their representations transfer poorly to dense prediction tasks (e.g., object detection and instance segmentation), which require precise object localization, recognition, and scale consistency.

Existing solutions attempt either pixel-level contrastive matching or object-level contrastive learning:

  • Pixel-level methods rely on approximate feature matching (like Hungarian matching or optimal transport), which can be noisy and imprecise.
  • Object-level methods rely on unsupervised proposals (e.g., Selective Search), which produce low-quality bounding boxes and masks on complex, non-iconic scenes.
  • Compute inefficiency: Typical SSL methods require 400 to 1,000+ epochs to converge, far exceeding the standard 90-epoch budget of supervised ImageNet pre-training.

Methods

MCL closes the gap between self-supervised pretext tasks and downstream dense prediction architectures via two main components:

  1. Montage Assembly: The authors downsample augmented images by factors of $2^s$ (where $s \in \{0, 1, 2, \dots, S-1\}$) and stitch them together into a non-overlapping mosaic grid matching the original image dimensions. Because the positions and scales are known by construction, the model receives exact pseudo-bounding boxes with zero label noise.
  2. Multi-Level Contrastive Learning:
  3. Sub-image coordinates are mapped directly to corresponding levels of a Feature Pyramid Network (FPN), where features are aggregated via RoI pooling and global average pooling.
  4. Using an online encoder and a momentum target encoder (updated via exponential moving average), MCL applies an InfoNCE loss across pyramid levels.
  5. The best-performing matching strategy contrasts all smaller-scale views against the view with the largest resolution, pulling representations toward high-resolution semantic features.
  6. End-to-End Task Alignment: Rather than pre-training only the backbone, MCL pre-trains the backbone, FPN neck, and detection head together.
Augmented Views (Multi-scale) 
       │
       ▼
 [Montage Grid]  ──► [Backbone + FPN] ──► [RoI Pool / Heads] ──► [Multi-Level Loss]
(Exact Bounding Boxes)                         (P3, P4, P5)       (Target Largest Scale)

Datasets

  • Pre-training Datasets: ImageNet-1K (~1.28M images) and COCO (train2017 + unlabeled2017, ~241k images).
  • Downstream Evaluation Benchmarks:
  • MS COCO: Bounding box detection, instance segmentation, keypoint detection, and low-data regimes (1%, 2%, 5%, 10% splits).
  • LVIS v1: Long-tail object detection and instance segmentation.
  • Cityscapes: Semantic and instance segmentation.
  • ADE20K-847: Semantic segmentation. (No new datasets were introduced.)

Key Insights and Ideas

  • Synthesizing multi-object scenes beats heuristic proposals: Constructing synthetic montages explicitly forces the network to handle scale variation and spatial localization with mathematically exact coordinates, avoiding the noisy proposals of Selective Search.
  • Cross-scale alignment anchors small objects to rich semantics: Contrasting low-resolution sub-regions against the highest-resolution target view yielded the best performance, proving that lower pyramid levels benefit from guidance provided by high-capacity, fine-grained representations.
  • Pre-training full detector heads matters: Ablations demonstrate that pre-training the FPN neck and detection head alongside the backbone boosts Mask R-CNN performance from $41.1$ to $42.5\ \text{AP}^\text{bb}$.
  • Pretext task generalizes to supervised training: Adapting the montage task to supervised pre-training also improves performance on downstream tasks for both CNNs (ResNet-50) and Vision Transformers (Swin-T), highlighting that task alignment is beneficial regardless of whether supervision is used.

Key Results

  • COCO 1× Mask R-CNN (ResNet-50): With only 100 ImageNet pre-training epochs, MCL achieves $42.5\ \text{AP}^\text{bb}$ and $38.2\ \text{AP}^\text{mk}$, surpassing supervised pre-training by $+3.6\ \text{AP}^\text{bb}$ / $+2.8\ \text{AP}^\text{mk}$ and outperforming 200-epoch MoCo by $+4.0\ \text{AP}^\text{bb}$.
  • COCO 1× RetinaNet: Achieves $39.1\ \text{AP}^\text{bb}$, outperforming the supervised baseline ($37.4$) and prior dense SSL methods like PixPro ($37.9$) and SoCo ($38.2$).
  • Non-Iconic Pre-training (COCO-pre-trained): Reaches $41.8\ \text{AP}^\text{bb}$ and $37.7\ \text{AP}^\text{mk}$, demonstrating robustness when pre-trained directly on scene-centric images rather than object-centric ImageNet data.
  • Low-Data Regime: When finetuning on 1% to 10% subsets of COCO, MCL consistently outperforms supervised baselines by around $+4\ \text{AP}$.
  • LVIS Long-Tail Detection: Improves detection on rare categories by $+4.3\ \text{AP}^\text{bb}_r$ and $+4.3\ \text{AP}^\text{mk}_r$ over the supervised baseline.

Limitations and Future Work

  • Boundary Artifacts: Tiling separate images introduces sharp, artificial boundaries. Although experiments showed that Gaussian boundary smoothing degraded performance (likely by corrupting edge-adjacent features), the paper relies on average pooling to buffer edge effects rather than natively modeling boundary transitions.
  • Rigid Grid Layout: Sub-images are placed into fixed regular grids ($1\times1$, $2\times2$, $4\times4$ downsamplings), which does not fully capture arbitrary aspect ratios, arbitrary occlusions, or natural object overlaps found in real-world scenes.
  • Future Directions: The authors suggest extending this framework to more fine-grained representation learning within sub-regions.

Summary updated

Abstract

In this work, we present Multi-Level Contrastive Learning for Dense Prediction Task (MCL), an efficient self-supervised method for learning region-level feature representation for dense prediction tasks. Our method is motivated by the three key factors in detection: localization, scale consistency and recognition. To explicitly encode absolute position and scale information, we propose a novel pretext task that assembles multi-scale images in a montage manner to mimic multi-object scenarios. Unlike the existing image-level self-supervised methods, our method constructs a multi-level contrastive loss that considers each sub-region of the montage image as a singleton. Our method enables the neural network to learn regional semantic representations for translation and scale consistency while reducing pre-training epochs to the same as supervised pre-training. Extensive experiments demonstrate that MCL consistently outperforms the recent state-of-the-art methods on various datasets with significant margins. In particular, MCL obtains 42.5 AP$^\mathrm{bb}$ and 38.3 AP$^\mathrm{mk}$ on COCO with the 1x schedule fintuning, when using Mask R-CNN with R50-FPN backbone pre-trained with 100 epochs. In comparison to MoCo, our method surpasses their performance by 4.0 AP$^\mathrm{bb}$ and 3.1 AP$^\mathrm{mk}$. Furthermore, we explore the alignment between pretext task and downstream tasks. We extend our pretext task to supervised pre-training, which achieves a similar performance to self-supervised learning. This result demonstrates the importance of the alignment between pretext task and downstream tasks, indicating the potential for wider applicability of our method beyond self-supervised settings.
BibTeX
@misc{guo2023multi,
  author = {Qiushan Guo and Yizhou Yu and Yi Jiang and Jiannan Wu and Zehuan Yuan and Ping Luo},
  title = {{Multi-Level Contrastive Learning for Dense Prediction Task}},
  year = {2023},
  eprint = {2304.02010},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.02010},
}

Loading related content...

Figure