Self-supervised Learning of Contextualized Local Visual Embeddings

Thalles Silva, Hélio Pedrini, Adín Ramírez Rivera

Abstract

Self-supervised learning (SSL) has revolutionized visual representation learning, yet most frameworks focus on global image-level features, often at the expense of local spatial information. This paper introduces CLoVE (Contextualized Local Visual Embeddings), a framework designed to bridge the gap between global context and local detail for dense prediction tasks such as object detection and semantic segmentation. By leveraging a specialized self-attention mechanism, CLoVE learns local embeddings that are informed by their semantic neighborhood. Our approach achieves state-of-the-art performance for CNN backbones across multiple benchmarks, including COCO, LVIS, and Cityscapes.

Introduction

Dominant SSL methods like SimCLR and BYOL typically optimize for image classification by collapsing an entire image into a single global vector. While effective for identifying object categories, this process discards the fine-grained spatial information necessary for "where" tasks—locating objects and defining their boundaries. Conversely, existing local SSL methods that match individual pixels or patches often lack the broader semantic context required for robust feature representation.

CLoVE addresses this trade-off by learning contextualized local embeddings. Instead of treating local points in isolation, CLoVE uses a self-attention layer to allow local features to "talk" to semantically similar regions nearby, creating a representation that is both spatially precise and contextually aware.

Conceptual comparison of SSL strategies: Global (top) collapses the image; Local (middle) matches isolated patches; CLoVE (bottom) uses contextualized embeddings to relate similar regions.

Methodology

The CLoVE framework enhances the standard student-teacher SSL pipeline with several key innovations:

Normalized Multi-Head Self-Attention (NMHSA)

The core of CLoVE is the NMHSA layer. Unlike traditional local SSL which predicts features directly, the student network in CLoVE employs attention to aggregate features from various parts of the image based on their similarity. This allows the model to recognize that disparate parts of an object (e.g., different parts of a horse's body) are related, contributing to a unified contextualized representation. Crucially, normalizing the queries and keys within this layer was found to be essential for training stability.

Neighborhood Matching and Sampling

To define positive pairs across augmented views, CLoVE utilizes a spatial locality bias. It matches points that are physically proximal in the original pixel space, rather than requiring exact pixel-level alignment. To further refine the feature space, CLoVE introduces Intra-Negative Sampling, identifying semantically distinct regions within the same image to serve as negative examples.

Ranking Margin Loss

The model is optimized using a ranking margin loss that maximizes the agreement between the student's contextualized predictions and the teacher's local targets while simultaneously pushing away the intra-negatives.

Experimental Results

CLoVE was pre-trained on ImageNet-1M and evaluated on a variety of downstream dense prediction tasks.

Object Detection and Segmentation

On the COCO benchmark using a ResNet-50 backbone, CLoVE achieved 41.2 AP for detection and 37.1 AP for segmentation, significantly outperforming previous state-of-the-art CNN-based SSL methods like PixPro and DenseCL.

Long-tail and Urban Scene Benchmarks

The framework demonstrated strong generalization on the LVIS dataset, improving upon the supervised baseline by +4 points across all metrics. On Cityscapes, CLoVE showed a substantial improvement of +1.4 AP over PixPro and a remarkable +10.7 AP over the standard supervised baseline.

Discussion and Conclusion

CLoVE demonstrates that contextualization is superior to simple locality for dense vision tasks. By integrating self-attention into standard CNN architectures during the self-supervised phase, the model learns richer, more discriminative features.

While CLoVE sets a new standard for CNN-based SSL, it is sensitive to the distance threshold ($T_{pos}$) used for neighborhood matching. Future work may explore more adaptive matching strategies and the scalability of contextualized embeddings to pure Vision Transformer (ViT) architectures.

Summary updated

Abstract

We present Contextualized Local Visual Embeddings (CLoVE), a self-supervised convolutional-based method that learns representations suited for dense prediction tasks. CLoVE deviates from current methods and optimizes a single loss function that operates at the level of contextualized local embeddings learned from output feature maps of convolution neural network (CNN) encoders. To learn contextualized embeddings, CLoVE proposes a normalized mult-head self-attention layer that combines local features from different parts of an image based on similarity. We extensively benchmark CLoVE’s pre-trained representations on multiple datasets. CLoVE reaches state-of-the-art performance for CNN-based architectures in 4 dense prediction downstream tasks, including object detection, instance segmentation, keypoint detection, and dense pose estimation. Code: https://github.com/sthalles/CLoVE.
BibTeX
@inproceedings{silva2023self,
  author = {Thalles Silva and Hélio Pedrini and Adín Ramírez Rivera},
  title = {{Self-supervised Learning of Contextualized Local Visual Embeddings}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  url = {https://openreview.net/forum?id=l8zZ5impsP},
}
Figure