Self-Supervised Learning With Local Contrastive Loss for Detection and Semantic Segmentation

Ashraful Islam, Benjamin Lundell, Harpreet Sawhney, Sudipta N. Sinha, Peter Morales, Richard J. RadkeWACV 2023

TL;DR

This paper introduces a simple yet effective "Local Contrastive Loss" (LC-loss) that can be easily integrated into existing self-supervised learning (SSL) frameworks. By enforcing pixel-level feature consistency between transformed image pairs, their method significantly boosts performance on downstream tasks like object detection and semantic segmentation, outperforming prior state-of-the-art approaches.

What Problem Does This Paper Address?

Self-supervised learning (SSL) has shown great success in learning general visual representations, particularly for image classification tasks. However, many existing SSL methods focus on learning global image features, which often discard crucial spatial information. This makes them less effective for "dense prediction" tasks like object detection and semantic segmentation, which require understanding fine-grained spatial relationships within an image. The challenge is to adapt SSL to retain and leverage this spatial information without relying on labeled data.

Methods

The core contribution is the Local Contrastive Loss (LC-loss). This loss function encourages corresponding pixels in two differently transformed versions of the same image to have similar learned features.

Here's how it works:

  • Leveraging Existing SSL: The LC-loss is designed to be an additive component to existing SSL frameworks. The authors primarily use BYOL (Bootstrap Your Own Latent) as their base SSL method, which is known for its strong performance without needing negative pairs.
  • Image Transformations: Two versions of an input image are created: one with color augmentations only (Ic) and another with both spatial (random resize crop, flip) and color transformations (Isc).
  • Dense Feature Extraction: Both online and target networks (from BYOL) extract dense, low-resolution feature maps.
  • Pixel Correspondence: Crucially, the spatial transformations applied to create Isc allow for known correspondences between pixels in Ic and Isc.
  • Contrastive Loss Calculation: A dense correspondence map is computed between the feature maps. The LC-loss then penalizes the model if the feature of a pixel in Ic does not have a high similarity score with its corresponding pixel in Isc. This is formulated as a negative log-likelihood, encouraging the model to maximize the similarity for true correspondences.
  • Combined Loss: The final loss is a weighted sum of the global SSL loss (e.g., BYOL's loss) and the new LC-loss.

What's novel is the simplicity: it doesn't require complex architectural changes like encoder-decoder networks for correspondence or explicit region proposal generation.

Datasets

  • Pretraining: ImageNet dataset (~1.28M images) was used for pretraining the backbone networks.
  • Downstream Tasks:
    • COCO (Common Objects in Context) for object detection and instance segmentation.
  • PASCAL VOC (Visual Object Classes) for object detection and semantic segmentation.
  • CityScapes for semantic segmentation.

The paper does not introduce new datasets.

Key Insights and Ideas

  • Spatial Information is Key for Dense Tasks: Global SSL methods are inherently limited for tasks requiring spatial understanding. Explicitly enforcing local consistency is a direct way to address this.
  • Simplicity Wins: The LC-loss is remarkably effective despite its simplicity. It doesn't require complex machinery like pixel-propagation modules or carefully tuned negative pairs, which were needed in prior works.
  • Complementary Losses: Combining a global SSL loss (like BYOL) with a local contrastive loss provides a synergistic effect, leading to better overall representations.
  • Robustness to Transformations: The LC-loss helps the model learn more robust feature representations that generalize well across different datasets and tasks.

Key Results

The proposed method, "OURS," consistently outperforms existing state-of-the-art SSL approaches on various downstream tasks:

  • COCO Object Detection: Achieved 40.6 AP, outperforming the previous best (PixPro) by 1.9%.
  • PASCAL VOC Object Detection: Achieved 60.1 AP, improving by 1.4% over the previous best.
  • CityScapes Segmentation: Achieved 77.8% mIoU, a 0.6% improvement over PixPro.
  • Frozen Backbone Evaluation: Showed significant gains, indicating strong representation quality. For instance, 30.5% AP on COCO object detection (2.8% over PixPro) and 63.4% mIoU on VOC segmentation (2.9% over PixPro).
  • Efficient Pre-training: The method achieves comparable or better results with significantly less pre-training time and fewer epochs compared to BYOL.

Limitations and Future Work

  • Low Spatial Resolution: The LC-loss is computed at a down-sampled resolution (32x for ResNet50). Computing it at higher resolutions would be more computationally expensive but might yield further improvements.
  • Uniform Sampling: Local correspondences are sampled on a uniform grid, which might not be optimal for capturing features at salient points (like corners) and could be too strict for texture-less regions.
  • Self-Similar Regions: The loss doesn't explicitly account for self-similar regions within an image, which could be a direction for future exploration.

The authors suggest that exploring higher resolutions and more sophisticated sampling strategies could be beneficial. They also demonstrate that their approach can generalize to other SSL methods like DINO.

Summary updated

BibTeX
@inproceedings{islam2023self,
  author = {Ashraful Islam and Benjamin Lundell and Harpreet Sawhney and Sudipta N. Sinha and Peter Morales and Richard J. Radke},
  title = {{Self-Supervised Learning With Local Contrastive Loss for Detection and Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure