No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

Walter Simoncini, Spyros Gidaris, Andrei Bursuc, Yuki M. Asano

TL;DR

FUNGI (Features from UNsupervised GradIents) is a "plug-and-play" method that boosts the performance of frozen, pretrained transformer models without any additional training. By concatenating standard model embeddings with projected gradients from self-supervised objectives (like DINO or SimCLR), the authors achieve significant gains in classification, retrieval, and scene understanding across vision, text, and audio domains.

What Problem Does This Paper Address?

Deep learning typically follows a "pretrain then fine-tune" paradigm. While pretrained backbones provide excellent general-purpose features, their frozen embedding spaces are often suboptimal for specific downstream tasks. Fine-tuning solves this but is computationally expensive and requires labeled data.

On the other hand, non-parametric methods like $k$-Nearest Neighbors ($k$NN) are efficient and require no training, but they are entirely dependent on the quality of the frozen representation. The authors ask: Can we extract more "adaptive" information from a frozen model for a specific data point without actually updating the weights?

Methods

The core idea of FUNGI is to treat the gradient of a self-supervised loss as a feature itself.

Figure 1: Gradient-augmented features: given a pretrained backbone fθ∗and its embeddings, we apply a family of SSL losses, extract their gradients, and project and concatenate them. These new features are used to build a k-nearest neighbor index, which can be used for classification or retrieval.

The process works as follows:

  1. Compute SSL Losses: For a given input (e.g., an image), the model calculates several self-supervised learning (SSL) losses. The paper uses three families: DINO (distillation/clustering), SimCLR (contrastive learning), and a KL-divergence loss (entropy maximization).
  2. Extract Gradients: Instead of updating weights, the method performs a backward pass to extract the per-sample gradients with respect to a specific layer (typically the last attention output projection).
  3. Dimensionality Reduction: Gradients are massive. To make them usable for retrieval, FUNGI uses binary random projections to squash these high-dimensional gradients down to the size of the original embedding.
  4. Feature Fusion: The original embedding and the projected gradients are concatenated and then further compressed using PCA to maintain a constant feature size (iso-storage).

This approach essentially captures "first-order learning signals"—how the model would change to adapt to that specific input—and uses that information to enrich the representation.

Figure 4: Gradients extraction using a SimCLR loss. Given a pretrained backbone f and a randomly initialized projection head h, we first patchify an image, obtain the latent representations of patches (1), calculate the SimCLR loss by maximizing the pairwise cosine similarity of patches, and minimizing their similarity to a fixed negatives batch and backpropagate (2), extract the per-sample gradients (3) and finally project the gradients to the same dimensionality as the embeddings (4).

Datasets

The authors conducted an extensive evaluation across multiple modalities:

  • Vision: 11 datasets including ImageNet-1K, CIFAR-10/100, Oxford Flowers 102, CUB-200, and Stanford Cars.
  • Text: 5 datasets including TREC (question classification), Banking-77, and AG News.
  • Audio: ESC-50 (environmental sounds) and SpeechCommands V2.
  • Dense Prediction: Pascal VOC 2012 and ADE20K for semantic segmentation.

Key Insights and Ideas

  • Gradients are Complementary: The "aha moment" is that gradients from different SSL objectives capture fundamentally different information than the standard embedding. As shown in the paper, these features have low similarity to each other (measured by CKA), meaning their combination provides a much richer description of the data.
  • Diversity Wins: Combining multiple types of gradients (e.g., KL + DINO + SimCLR) consistently outperforms using just one.
  • Deeper is Better: Gradients extracted from the final layers of a transformer are significantly more predictive than those from earlier layers, mirroring how semantic information is built up in the forward pass.
  • Zero-Shot Adaptation: FUNGI acts as a form of non-parametric transfer learning. It allows a frozen model to "sense" the local curvature of the loss landscape for new data, providing a task-specific signal without a single weight update.
Figure 2: Combining diverse features (em- beddings, gradients) leads to large improve- ments. Pairwise CKA similarity of features (top) and the kNN accuracy of their combina- tion (bottom).

Key Results

FUNGI delivers consistent improvements across 20 different backbones (ViT, DeIT, CLIP, DINOv2, etc.):

  • Semantic Segmentation: Using FUNGI features improved a frozen DINO model by a staggering +17% mIoU on Pascal VOC without any training.
  • Classification: On average, FUNGI improved $k$NN accuracy by 4.4% on the full dataset and 2.8% in 5-shot scenarios across 11 vision benchmarks.
  • Data Efficiency: The method is particularly potent in low-data regimes, showing significant boosts when only 3 to 6 examples per class are available.
  • Multimodal Success: The benefits extended to BERT and T5 for text, and SSAST for audio, proving the method's generality.
Performance gains are consistent across various backbone architectures and sizes.

Limitations and Future Work

  • Computational Cost: The main drawback is the overhead. Computing gradients requires a backward pass, which is slower than a simple forward pass. For example, SimCLR gradients can be ~40x slower to compute than standard embeddings due to the multiple views required.
  • Diminishing Returns with Scale: The authors noted that as the pretraining dataset size grows (e.g., moving from ImageNet-1K to 2B image-text pairs), the relative gain from FUNGI decreases, likely because the base embeddings are already highly optimized.
  • Text Augmentation: The SimCLR objective for text relied on simple word deletion; the authors suggest that more sophisticated augmentations (like back-translation) might yield even better text gradients.

In summary, FUNGI proves that there is a wealth of untapped information hidden in the gradients of frozen models, offering a powerful new tool for researchers working with fixed feature extractors.

Summary updated

Abstract

This paper introduces FUNGI, Features from UNsupervised GradIents, a method to enhance the features of transformer encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These gradients are projected to a lower dimension and then concatenated with the model's output embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification, clustering and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training.
BibTeX
@inproceedings{simoncini2024no,
  author = {Walter Simoncini and Spyros Gidaris and Andrei Bursuc and Yuki M. Asano},
  title = {{No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=PRBsEz8rnV},
}
Figure