No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations

Walter Simoncini, Andrei Bursuc, Spyros Gidaris, Yuki M Asano

In the world of deep learning, the standard "recipe" for using a pretrained model is either to use its output embeddings as-is (frozen) or to fine-tune it on a specific task. Fine-tuning is powerful but expensive and requires labeled data. Using frozen embeddings is fast but often leaves performance on the table.

A new paper from researchers at the University of Amsterdam and valeo.ai introduces a third way. Their method, FUNGI (Features from UNsupervised GradIents), enhances frozen representations without a single step of actual training. The secret ingredient? Using the model's own gradients as features.

TL;DR

FUNGI improves the performance of frozen transformer models by concatenating their standard output embeddings with gradients derived from self-supervised "pretext" tasks. This training-free approach provides consistent gains across vision, text, and audio, even boosting semantic segmentation performance by up to 17% over a standard DINO baseline.

The Problem: Extracting More from Frozen Models

When we use a "frozen" backbone (like a Vision Transformer or BERT), we are essentially treating the model as a fixed feature extractor. However, these fixed features might not capture the specific nuances of a new dataset.

While we could fine-tune the model, we often lack the labels or the compute to do so. The authors ask: Can we extract more information from a frozen model than just its final output? They hypothesize that the gradients—the signals telling the model how it should change to solve a task—contain rich, complementary information that the final embeddings miss.

The FUNGI Method

The FUNGI pipeline is surprisingly "plug-and-play" and requires no human labels or weight updates. It simply "interrogates" the frozen model more deeply.

The FUNGI pipeline extracts gradients from multiple self-supervised losses, projects them, and concatenates them with the standard embedding.
  1. Compute Self-Supervised Losses: For any given input, the authors compute several self-supervised learning (SSL) losses on top of the frozen backbone. They use three main types:
  2. DINO: A distillation loss based on global/local crop correspondence.
  3. SimCLR: A contrastive loss that compares different patches of the same image.
  4. KL Divergence: A loss that encourages the model's output distribution to be uniform (maximizing entropy).
  5. Extract Gradients: Instead of updating the weights, they simply record the gradients of these losses with respect to a specific layer (usually the last attention projection).
  6. Dimensionality Reduction: Gradients are massive. To make them usable, FUNGI projects them into a lower-dimensional space using Binary Random Projections (a fast, hardware-friendly technique).
  7. Concatenate and Compress: These projected gradients are concatenated with the original model embedding. Finally, PCA is used to shrink this combined vector back down to the original embedding size, ensuring that downstream tasks (like k-Nearest Neighbors) don't slow down.

Key Insights and Ideas

  • Gradients are Complementary: Using Centered Kernel Alignment (CKA), the authors show that gradient features and embedding features are quite different. Combining them provides a "best of both worlds" representation.
  • The "Aha" Moment for Scene Understanding: The most striking result is in "in-context" semantic segmentation. By using FUNGI features to retrieve similar patches from a memory bank, a standard DINO model can suddenly perform dense segmentation with high accuracy, rivaling models specifically trained for the task.
  • Deeper is Better: The authors found that gradients from the deepest layers of the transformer are significantly more "predictive" than those from earlier layers.
  • Universal Applicability: Because SSL objectives exist for almost every modality, FUNGI works across vision, text, and audio transformers without changing the core logic.

Key Results

The authors conducted a massive evaluation across three modalities, including 11 vision datasets, 5 NLP datasets, and 2 audio datasets.

FUNGI consistently improves accuracy across a wide range of vision transformer backbones compared to standard embeddings.
  • Consistent Wins: FUNGI improved k-Nearest Neighbor (kNN) classification across 20 different backbones (including DINOv2, CLIP, and MAE), yielding consistent accuracy boosts on full datasets.
  • Massive Segmentation Boost: On the Pascal VOC dataset, FUNGI improved DINO ViT-B/16's semantic segmentation score from 55.7 to 67.0 mIoU (+11.3 points) without any training.
  • Data Efficiency: In "few-shot" settings (where you only have 1–5 examples per class), FUNGI provides a significant safety net, often outperforming standard embeddings by a wide margin.
  • Text & Audio: It improved intent classification in Banking-77 by 2.5% and boosted audio classification on ESC-50 from 42.8% to 47.0%.

Limitations and Future Work

While FUNGI doesn't require training, it does require a "backward pass" to compute gradients for every input. This makes feature extraction slower (e.g., DINO gradients are ~15x slower to compute than just the forward pass). Additionally, the authors noted "diminishing returns" on models trained on gargantuan datasets (like EVA-CLIP, trained on 2 billion images), as these models are already so well-optimized that the SSL gradients provide less "new" information.

Future directions include exploring more efficient ways to compute these gradients and investigating if FUNGI can help detect "out-of-distribution" data or improve the reliability of Retrieval-Augmented Generation (RAG) systems.

Paper Citation: Walter Simoncini, Spyros Gidaris, Andrei Bursuc, Yuki M. Asano. "No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations." NeurIPS 2024. Code available on GitHub.

Summary updated

Abstract

This paper introduces FUNGI, **F**eatures from **UN**supervised **G**rad**I**ents, a method to enhance the features of transformer encoders by leveraging self-supervised gradients. Our method is simple: given any pretrained model, we first compute gradients from various self-supervised objectives for each input. These gradients are projected to a lower dimension and then concatenated with the model's output embedding. The resulting features are evaluated on k-nearest neighbor classification over 11 datasets from vision, 5 from natural language processing, and 2 from audio. Across backbones spanning various sizes and pretraining strategies, FUNGI features provide consistent performance improvements over the embeddings. We also show that using FUNGI features can benefit linear classification, clustering and image retrieval, and that they significantly improve the retrieval-based in-context scene understanding abilities of pretrained models, for example improving upon DINO by +17% for semantic segmentation - without any training. Code is available at https://github.com/WalterSimoncini/fungivision.
BibTeX
@inproceedings{simoncini2024no,
  author = {Walter Simoncini and Andrei Bursuc and Spyros Gidaris and Yuki M Asano},
  title = {{No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=PRBsEz8rnV},
}
Figure