Mean Shift for Self-Supervised Learning

Soroush Abbasi Koohpayegani, Ajinkya Tejankar, Hamed Pirsiavash

Abstract

Most recent self-supervised learning (SSL) algorithms learn features by contrasting between instances of images or by clustering the images and then contrasting between the image clusters. We introduce a simple mean-shift algorithm that learns representations by grouping images together without contrasting between them or adopting much of prior on the structure of the clusters. We simply "shift" the embedding of each image to be close to the "mean" of its neighbors. Since in our setting, the closest neighbor is always another augmentation of the same image, our model will be identical to BYOL when using only one nearest neighbor instead of 5 as used in our experiments. Our model achieves 72.4% on ImageNet linear evaluation with ResNet50 at 200 epochs outperforming BYOL. Our code is available here: https://github.com/UMBCvision/MSF

Introduction

Self-supervised learning (SSL) has revolutionized computer vision by enabling models to learn powerful representations from unlabeled data. Current state-of-the-art methods typically fall into two categories: contrastive learning and clustering. Contrastive methods, such as SimCLR or MoCo, learn by pulling different augmentations of the same image together while pushing away "negative" samples. However, these methods often require large batch sizes or memory banks and can suffer from "class collision," where semantically similar images are incorrectly treated as negatives. Clustering methods, like SwAV, avoid explicit negatives by assigning images to prototypes, but they often impose rigid structural priors on the data.

Mean Shift for Self-Supervised Learning (MSF) offers a middle ground. It groups semantically similar images locally and fluidly without requiring negative samples or explicit cluster assignments. By shifting an image's embedding toward the mean of its nearest neighbors, MSF achieves superior performance and robustness, particularly in scenarios where data augmentation is limited.

The MSF Framework

MSF is built upon the Bootstrap Your Own Latent (BYOL) architecture, which utilizes an online encoder and a target encoder (a moving average of the online encoder). The core innovation of MSF is the integration of a nearest-neighbor search within the target calculation process.

The MSF architecture (top) vs. BYOL (bottom). MSF pulls the query embedding toward the mean of $k$ nearest neighbors from a memory bank, whereas BYOL pulls toward a single augmented view.

The process follows four primary steps:

  1. Dual Views: An unlabeled image is processed through two different augmentation pipelines ($T_1$ and $T_2$).
  2. Memory Bank: The target encoder processes one view, and the resulting embedding is stored in a large, first-in-first-out (FIFO) memory bank containing embeddings of previously seen images.
  3. Nearest Neighbor Search: For a given target embedding, the model identifies its $k$ nearest neighbors (NNs) within the memory bank.
  4. Mean Shift Loss: The online encoder processes the second view. The model is then trained to minimize the distance between this online embedding and the mean (average) of the $k$ nearest neighbors identified in the previous step.

When $k=1$, MSF is equivalent to BYOL. By increasing $k$ (typically to 5), the model learns from a broader neighborhood of semantically similar images, effectively treating these neighbors as "super-augmentations" that provide natural variations a simple crop or color shift cannot.

Analysis and Results

MSF demonstrates significant improvements over baseline SSL methods, particularly in its ability to learn from weak signals and its robustness to varied augmentation strategies.

Bootstrapping Semantic Similarity

A remarkable property of MSF is its ability to "bootstrap" semantic understanding. At the beginning of training, the nearest neighbors retrieved from the memory bank are largely random, with an accuracy of only about 1.5%. However, even this very weak initial signal is sufficient for the model to begin organizing the feature space.

Nearest neighbors (NN) retrieved from the memory bank at different training stages. Initially random, the neighbors become semantically consistent (e.g., golden retrievers) as the model learns.

As shown in the visualization above, by Epoch 200, the retrieved neighbors for a query image (e.g., a golden retriever) are almost exclusively images of the same semantic class, despite the model never seeing a label. This suggests that the model successfully learns to group images based on high-level semantic content rather than low-level pixel similarities.

Performance Benchmarks

On the ImageNet linear evaluation benchmark, MSF with a ResNet50 backbone achieves 72.4% top-1 accuracy after 200 epochs. This outperforms BYOL (70.6%) and other SOTA methods like MoCo v2 and SwAV under similar constraints. Despite the nearest neighbor search, the computational overhead is minimal; using a 128K memory bank adds less than 1% to total computation time.

Robustness to Weak Augmentation

One of the most compelling advantages of MSF is its performance under "weak" data augmentation (e.g., only random cropping and horizontal flipping). While standard contrastive methods like SimCLR see a massive performance drop (plummeting to 40.2%) without aggressive color jittering and blurring, MSF maintains a high accuracy of 66.3%. This robustness makes MSF particularly suitable for domains like medical imaging, where preserving original color and texture information is critical for diagnostic accuracy.

Conclusion

Mean Shift for Self-Supervised Learning provides a simple, effective, and efficient framework for representation learning. By leveraging the local structure of the feature space through nearest neighbors, it avoids the pitfalls of contrastive negatives and rigid clustering. Future work may explore dynamic $k$ values that increase as training progresses, potentially further enhancing the model's ability to capture complex semantic relationships as the feature space becomes more organized.

Summary updated

Abstract

Most recent self-supervised learning (SSL) algorithms learn features by contrasting between instances of images or by clustering the images and then contrasting between the image clusters. We introduce a simple mean-shift algorithm that learns representations by grouping images together without contrasting between them or adopting much of prior on the structure of the clusters. We simply "shift" the embedding of each image to be close to the "mean" of its neighbors. Since in our setting, the closest neighbor is always another augmentation of the same image, our model will be identical to BYOL when using only one nearest neighbor instead of 5 as used in our experiments. Our model achieves 72.4% on ImageNet linear evaluation with ResNet50 at 200 epochs outperforming BYOL. Our code is available here: https://github.com/UMBCvision/MSF
BibTeX
@inproceedings{koohpayegani2021mean,
  author = {Soroush Abbasi Koohpayegani and Ajinkya Tejankar and Hamed Pirsiavash},
  title = {{Mean Shift for Self-Supervised Learning}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=vt51uUyfVMO},
}
Figure