Abstract
This paper introduces a novel approach to improving the training stability of self-supervised learning (SSL) methods by leveraging a non-parametric memory of seen concepts. The proposed method involves augmenting a neural network with a memory component to stochastically compare current image views with previously encountered concepts. Additionally, we introduce stochastic memory blocks to regularize training and enforce consistency between image views. We extensively benchmark our method on many vision tasks, such as linear probing, transfer learning, low-shot classification, and image retrieval on many datasets. The experimental results consolidate the effectiveness of the proposed approach in achieving stable SSL training without additional regularizers while learning highly transferable representations and requiring less computing time and resources.
Introduction
Self-supervised learning (SSL) has transformed computer vision by enabling models to learn rich representations from vast quantities of unlabeled data. Among the most successful paradigms are clustering-based methods, which group similar images into "prototypes." However, these methods are notoriously difficult to train. Without the grounding of human labels, models often succumb to "training collapse," a failure mode where the network maps all inputs to a single cluster to minimize loss without learning meaningful features.
To combat collapse, current state-of-the-art methods like DINO or SwAV rely on complex engineering "band-aids," such as centering, target sharpening, or Sinkhorn-Knopp normalization. Furthermore, recent advancements like iBOT require processing every individual patch of an image, leading to significant computational overhead. This paper introduces MaSSL (Memory Augmented Self-Supervised Learning), a framework that replaces unstable gradient-based prototype learning with a non-parametric "working memory" of past experiences, resulting in a more stable and efficient training process.
Methodology
The core innovation of MaSSL is the transition from parametric to non-parametric anchors. Instead of optimizing a fixed set of cluster centers via gradient descent, MaSSL utilizes the model's own history to define the feature space.
Non-Parametric Memory
MaSSL maintains a simple First-In, First-Out (FIFO) queue that stores the most recent image embeddings generated by the model. These stored embeddings serve as "anchors" for comparison. By using actual seen images as anchors rather than abstract, learned prototypes, the model avoids the "moving target" problem that typically leads to mode collapse. This inherent stability allows MaSSL to function effectively without the complex regularizers required by its predecessors.
Stochastic Memory Blocks
Comparing a new image to a massive memory queue (often exceeding 65,000 entries) is computationally expensive and susceptible to "ordering bias," where the model over-fits to the most recent entries. MaSSL addresses this through Stochastic Memory Blocks. The memory is partitioned into random, disjoint subsets. During training, the model compares two different "views" (distorted crops) of the same image against these random blocks.
Consistency Task and Efficiency
The learning objective enforces consistency: both views of the same image must produce the same similarity pattern relative to the concepts stored in the memory blocks. Notably, MaSSL focuses its optimization on the global [CLS] token of the Vision Transformer (ViT) rather than every individual patch. This design choice drastically reduces the computational footprint, making MaSSL roughly 9% faster and requiring 25% less GPU memory than patch-intensive models like iBOT.
Experimental Results
The authors evaluated MaSSL across an extensive benchmarking suite, including ImageNet-1M pre-training and a variety of downstream tasks.
Transfer Learning and Robustness
On a suite of eight diverse transfer learning datasets (including Oxford-IIIT Pet, Stanford Cars, and GTSRB), MaSSL outperformed leading competitors like DINO and iBOT on five out of eight tasks. The model showed significant gains in specialized domains, such as a +2.6% improvement on FGVC Aircraft and +4.6% on GTSRB.
MaSSL also demonstrated superior performance in "low-shot" scenarios, learning effectively from as little as 1% of available labels, and showed higher resilience when training on "long-tailed" (highly unbalanced) datasets like ImageNet-LT.
Image Retrieval
In retrieval tasks—where the model must identify specific landmarks or objects within large databases (e.g., Oxford-5k, Paris-6k)—MaSSL’s ViT-S model consistently outperformed DINO. This suggests that the features learned through non-parametric memory are highly discriminative and generalize well to fine-grained visual search.
Discussion and Limitations
While MaSSL offers a more stable and efficient alternative to traditional SSL, the authors noted certain constraints. Performance gains tend to saturate once the memory queue reaches approximately 65,536 entries, suggesting that simply increasing memory size does not yield infinite improvements. Additionally, because MaSSL currently prioritizes global image features via the [CLS] token, it does not explicitly optimize for local patch-level features, though it still performs competitively on sparse correspondence tasks.
Conclusion
MaSSL demonstrates that non-parametric memory is a powerful tool for stabilizing self-supervised learning. By leveraging a stochastic comparison mechanism and a FIFO queue of past experiences, the model achieves state-of-the-art transferability and robustness while being significantly more resource-efficient than existing clustering-based methods. Future work may explore extending this memory-augmented approach to dense, patch-level tasks or developing more sophisticated memory management strategies beyond the FIFO architecture.
Summary updated