Abstract
Self-supervised learning (SSL) has revolutionized visual representation learning, yet most frameworks focus on global image-level features, often at the expense of local spatial information. This paper introduces CLoVE (Contextualized Local Visual Embeddings), a framework designed to bridge the gap between global context and local detail for dense prediction tasks such as object detection and semantic segmentation. By leveraging a specialized self-attention mechanism, CLoVE learns local embeddings that are informed by their semantic neighborhood. Our approach achieves state-of-the-art performance for CNN backbones across multiple benchmarks, including COCO, LVIS, and Cityscapes.
Introduction
Dominant SSL methods like SimCLR and BYOL typically optimize for image classification by collapsing an entire image into a single global vector. While effective for identifying object categories, this process discards the fine-grained spatial information necessary for "where" tasks—locating objects and defining their boundaries. Conversely, existing local SSL methods that match individual pixels or patches often lack the broader semantic context required for robust feature representation.
CLoVE addresses this trade-off by learning contextualized local embeddings. Instead of treating local points in isolation, CLoVE uses a self-attention layer to allow local features to "talk" to semantically similar regions nearby, creating a representation that is both spatially precise and contextually aware.
Methodology
The CLoVE framework enhances the standard student-teacher SSL pipeline with several key innovations:
Normalized Multi-Head Self-Attention (NMHSA)
The core of CLoVE is the NMHSA layer. Unlike traditional local SSL which predicts features directly, the student network in CLoVE employs attention to aggregate features from various parts of the image based on their similarity. This allows the model to recognize that disparate parts of an object (e.g., different parts of a horse's body) are related, contributing to a unified contextualized representation. Crucially, normalizing the queries and keys within this layer was found to be essential for training stability.
Neighborhood Matching and Sampling
To define positive pairs across augmented views, CLoVE utilizes a spatial locality bias. It matches points that are physically proximal in the original pixel space, rather than requiring exact pixel-level alignment. To further refine the feature space, CLoVE introduces Intra-Negative Sampling, identifying semantically distinct regions within the same image to serve as negative examples.
Ranking Margin Loss
The model is optimized using a ranking margin loss that maximizes the agreement between the student's contextualized predictions and the teacher's local targets while simultaneously pushing away the intra-negatives.
Experimental Results
CLoVE was pre-trained on ImageNet-1M and evaluated on a variety of downstream dense prediction tasks.
Object Detection and Segmentation
On the COCO benchmark using a ResNet-50 backbone, CLoVE achieved 41.2 AP for detection and 37.1 AP for segmentation, significantly outperforming previous state-of-the-art CNN-based SSL methods like PixPro and DenseCL.
Long-tail and Urban Scene Benchmarks
The framework demonstrated strong generalization on the LVIS dataset, improving upon the supervised baseline by +4 points across all metrics. On Cityscapes, CLoVE showed a substantial improvement of +1.4 AP over PixPro and a remarkable +10.7 AP over the standard supervised baseline.
Discussion and Conclusion
CLoVE demonstrates that contextualization is superior to simple locality for dense vision tasks. By integrating self-attention into standard CNN architectures during the self-supervised phase, the model learns richer, more discriminative features.
While CLoVE sets a new standard for CNN-based SSL, it is sensitive to the distance threshold ($T_{pos}$) used for neighborhood matching. Future work may explore more adaptive matching strategies and the scalability of contextualized embeddings to pure Vision Transformer (ViT) architectures.
Summary updated