Mean Shift for Self-Supervised Learning

Soroush Abbasi Koohpayegani, Ajinkya Tejankar, Hamed Pirsiavash

Abstract

Most recent self-supervised learning (SSL) algorithms learn features by contrasting between instances of images or by clustering the images and then contrasting between the image clusters. We introduce a simple mean-shift algorithm that learns representations by grouping images together without contrasting between them or adopting much of prior on the structure or number of the clusters. We simply "shift" the embedding of each image to be close to the "mean" of the neighbors of its augmentation. Since the closest neighbor is always another augmentation of the same image, our model will be identical to BYOL when using only one nearest neighbor instead of 5 used in our experiments. Our model achieves 72.4% on ImageNet linear evaluation with ResNet50 at 200 epochs outperforming BYOL. Also, our method outperforms the SOTA by a large margin when using weak augmentations only, facilitating adoption of SSL for other modalities.

1. Introduction

Self-supervised learning (SSL) has traditionally relied on two main paradigms: contrastive learning and clustering. Contrastive methods like MoCo or SimCLR pull different views of the same image together while pushing away "negative" samples. However, this can lead to false negatives where semantically similar images are pushed apart. Clustering methods like SwAV avoid this by grouping images into a fixed number of clusters, but often impose rigid priors on cluster shape and count.

We propose Mean Shift for Self-Supervised Learning (MSF), a framework that groups semantically similar images locally without the need for negative samples or global clustering constraints. MSF learns representations by "shifting" an image's embedding toward the average of its nearest neighbors in feature space, effectively performing local clustering on the fly.

2. Methodology

MSF builds upon the architecture of Bootstrap Your Own Latent (BYOL). It employs two neural networks: an online encoder, which is updated via gradients, and a target encoder, which is a slow-moving average of the online encoder.

The core innovation is the Nearest Neighbor (NN) search mechanism. For an input image, the model generates two augmented views. One view is processed by the target encoder to produce an embedding, which is stored in a memory bank of recent samples. Instead of aligning the online embedding solely with its own target view (as in BYOL), MSF identifies the $k$ nearest neighbors of that target view within the memory bank. The model then minimizes the distance between the online embedding and the mean of these $k$ neighbors.

Figure 1: The MSF architecture uses a target encoder and a memory bank to find nearest neighbors, shifting the online embedding toward their mean. This generalizes BYOL (where $k=1$) to a local mean-shift operation.

3. Experimental Results

We evaluated MSF on ImageNet and various transfer learning tasks. With a ResNet-50 backbone trained for 200 epochs, MSF achieved 72.4% top-1 accuracy on ImageNet linear evaluation, surpassing BYOL (70.6%) and MoCo v2 (67.5%).

A key finding is MSF's robustness to weak augmentations. While standard SSL methods often fail without aggressive color jittering or blurring, MSF maintains high performance (66.3% accuracy with only random crops and flips) because nearest neighbors provide semantic diversity that acts as a "super-augmentation." Finding a different image of the same category is a more powerful semantic signal than simple geometric transformations.

Figure 2: Visualization of the 13 nearest neighbors for a query image over 200 epochs. As training progresses, the neighbors transition from low-level visual similarity (Epoch 0) to high-level semantic consistency (Epoch 200).

4. Discussion and Conclusion

MSF demonstrates that explicit negative pairs are unnecessary for learning rich, separable features. By leveraging local neighborhood structure, the model effectively performs on-the-fly clustering without predefined cluster centers. Despite the nearest neighbor search, MSF adds less than 1% computational overhead compared to standard training. Future work may explore dynamic adjustments to $k$ during training to allow for coarser clustering as representations mature. MSF's robustness to weak augmentations makes it particularly promising for modalities like medical imaging, where aggressive data distortion is undesirable.

Summary updated

Abstract

Most recent self-supervised learning (SSL) algorithms learn features by contrasting between instances of images or by clustering the images and then contrasting between the image clusters. We introduce a simple mean-shift algorithm that learns representations by grouping images together without contrasting between them or adopting much of prior on the structure or number of the clusters. We simply "shift" the embedding of each image to be close to the "mean" of the neighbors of its augmentation. Since the closest neighbor is always another augmentation of the same image, our model will be identical to BYOL when using only one nearest neighbor instead of 5 used in our experiments. Our model achieves 72.4% on ImageNet linear evaluation with ResNet50 at 200 epochs outperforming BYOL. Also, our method outperforms the SOTA by a large margin when using weak augmentations only, facilitating adoption of SSL for other modalities. Our code is available here: https://github.com/UMBCvision/MSF
BibTeX
@inproceedings{koohpayegani2021mean,
  author = {Soroush Abbasi Koohpayegani and Ajinkya Tejankar and Hamed Pirsiavash},
  title = {{Mean Shift for Self-Supervised Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=vt51uUyfVMO},
}
Figure