No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

Walter Simoncini, Spyros Gidaris, Andrei Bursuc, Yuki M. Asano

Abstract

This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of vision encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These are projected to a lower dimension and then concatenated with the model's embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training.

Introduction

In modern machine learning, frozen pretrained models (such as CLIP or DINO) are ubiquitous as feature extractors. By feeding an input into a frozen backbone and extracting the final embedding, practitioners can perform downstream tasks like k-nearest neighbor (kNN) classification or image retrieval. However, these static embeddings are inherently limited: they are "blind" to the specific nuances of downstream data because the model weights remain fixed. While fine-tuning allows for adaptation, it is computationally expensive, requires labeled data, and can lead to catastrophic forgetting.

This work introduces FUNGI (Features from UNsupervised GradIents), a framework that makes frozen representations more expressive and adaptive at test-time without updating a single model weight. The core idea is to treat the gradient of a self-supervised loss function as a rich, first-order learning signal that complements standard embeddings.

The FUNGI Framework

The FUNGI pipeline: extracting and combining self-supervised gradients with frozen embeddings.

FUNGI operates by appending projected gradients to standard model embeddings. The process follows five key steps:

  1. Frozen Backbone: A pretrained transformer backbone (e.g., a ViT) is kept frozen, and a randomly initialized linear head is attached.
  2. Pseudo-Tasks: For every input, a self-supervised learning (SSL) loss is computed. The authors utilize three primary objectives: SimCLR (contrastive loss on patches), DINO (distillation based on local-to-global correspondences), and KL Divergence (forcing output distributions toward uniformity).
  3. Gradient Extraction: A backward pass computes the gradients of these losses with respect to the weights of the final transformer block.
  4. Dimensionality Reduction: To manage the massive size of raw gradients, Binary Random Projections are used to compress them to the same dimensionality as the original embedding.
  5. Feature Concatenation: The original embedding is concatenated with these projected gradients. PCA is optionally applied to maintain a manageable total feature size.

By capturing how the model would need to change to better understand a specific input, FUNGI provides an adaptive signal that static embeddings lack.

Empirical Evaluation

The authors evaluated FUNGI across 20 different backbones and multiple modalities, including vision (11 datasets), text (5 benchmarks), and audio (2 benchmarks).

Accuracy gains of FUNGI over standard embeddings across multiple backbones in full-dataset and few-shot settings.

Key Results

  • Consistent Improvements: FUNGI consistently outperformed standard embeddings across all tested backbones. For a ViT-B/16 pretrained on ImageNet-1K, FUNGI improved average kNN accuracy across 11 vision datasets by +4.4%.
  • Scene Understanding: The method showed dramatic success in retrieval-based semantic segmentation, improving DINO’s performance by +17% mIoU on Pascal VOC without any training.
  • Few-Shot Efficiency: Performance gains were particularly pronounced in few-shot scenarios, where labeled data is scarce, making FUNGI highly effective for niche domains.
  • Cross-Modal Versatility: The framework is not limited to vision; it improved BERT’s text classification on the Banking-77 dataset by +2.5%.

Analysis and Limitations

Analysis reveals that embeddings and gradients capture complementary information: embeddings represent what the model sees, while gradients represent the direction the model would take to solve a task. Combining multiple SSL objectives (e.g., SimCLR + DINO) yields better results than any single objective, suggesting these signals are often orthogonal.

However, the method introduces a significant computational overhead. Computing gradients requires a backward pass, which can be up to 40x slower than a standard forward pass. Additionally, the benefits of FUNGI show diminishing returns on models trained on massive, near-perfect datasets (like EVA-CLIP), where the gradient signal becomes less informative. Future work will focus on more efficient gradient computation and identifying new SSL objectives for specialized modalities.

Summary updated

Abstract

This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of vision encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These are projected to a lower dimension and then concatenated with the model's embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training.
BibTeX
@inproceedings{simoncini2024no,
  author = {Walter Simoncini and Spyros Gidaris and Andrei Bursuc and Yuki M. Asano},
  title = {{No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=PRBsEz8rnV},
}
Figure