In the world of deep learning, the standard "recipe" for using a pretrained model is either to use its output embeddings as-is (frozen) or to fine-tune it on a specific task. Fine-tuning is powerful but expensive and requires labeled data. Using frozen embeddings is fast but often leaves performance on the table.
A new paper from researchers at the University of Amsterdam and valeo.ai introduces a third way. Their method, FUNGI (Features from UNsupervised GradIents), enhances frozen representations without a single step of actual training. The secret ingredient? Using the model's own gradients as features.
TL;DR
FUNGI improves the performance of frozen transformer models by concatenating their standard output embeddings with gradients derived from self-supervised "pretext" tasks. This training-free approach provides consistent gains across vision, text, and audio, even boosting semantic segmentation performance by up to 17% over a standard DINO baseline.
The Problem: Extracting More from Frozen Models
When we use a "frozen" backbone (like a Vision Transformer or BERT), we are essentially treating the model as a fixed feature extractor. However, these fixed features might not capture the specific nuances of a new dataset.
While we could fine-tune the model, we often lack the labels or the compute to do so. The authors ask: Can we extract more information from a frozen model than just its final output? They hypothesize that the gradients—the signals telling the model how it should change to solve a task—contain rich, complementary information that the final embeddings miss.
The FUNGI Method
The FUNGI pipeline is surprisingly "plug-and-play" and requires no human labels or weight updates. It simply "interrogates" the frozen model more deeply.
- Compute Self-Supervised Losses: For any given input, the authors compute several self-supervised learning (SSL) losses on top of the frozen backbone. They use three main types:
- DINO: A distillation loss based on global/local crop correspondence.
- SimCLR: A contrastive loss that compares different patches of the same image.
- KL Divergence: A loss that encourages the model's output distribution to be uniform (maximizing entropy).
- Extract Gradients: Instead of updating the weights, they simply record the gradients of these losses with respect to a specific layer (usually the last attention projection).
- Dimensionality Reduction: Gradients are massive. To make them usable, FUNGI projects them into a lower-dimensional space using Binary Random Projections (a fast, hardware-friendly technique).
- Concatenate and Compress: These projected gradients are concatenated with the original model embedding. Finally, PCA is used to shrink this combined vector back down to the original embedding size, ensuring that downstream tasks (like k-Nearest Neighbors) don't slow down.
Key Insights and Ideas
- Gradients are Complementary: Using Centered Kernel Alignment (CKA), the authors show that gradient features and embedding features are quite different. Combining them provides a "best of both worlds" representation.
- The "Aha" Moment for Scene Understanding: The most striking result is in "in-context" semantic segmentation. By using FUNGI features to retrieve similar patches from a memory bank, a standard DINO model can suddenly perform dense segmentation with high accuracy, rivaling models specifically trained for the task.
- Deeper is Better: The authors found that gradients from the deepest layers of the transformer are significantly more "predictive" than those from earlier layers.
- Universal Applicability: Because SSL objectives exist for almost every modality, FUNGI works across vision, text, and audio transformers without changing the core logic.
Key Results
The authors conducted a massive evaluation across three modalities, including 11 vision datasets, 5 NLP datasets, and 2 audio datasets.
- Consistent Wins: FUNGI improved k-Nearest Neighbor (kNN) classification across 20 different backbones (including DINOv2, CLIP, and MAE), yielding consistent accuracy boosts on full datasets.
- Massive Segmentation Boost: On the Pascal VOC dataset, FUNGI improved DINO ViT-B/16's semantic segmentation score from 55.7 to 67.0 mIoU (+11.3 points) without any training.
- Data Efficiency: In "few-shot" settings (where you only have 1–5 examples per class), FUNGI provides a significant safety net, often outperforming standard embeddings by a wide margin.
- Text & Audio: It improved intent classification in Banking-77 by 2.5% and boosted audio classification on ESC-50 from 42.8% to 47.0%.
Limitations and Future Work
While FUNGI doesn't require training, it does require a "backward pass" to compute gradients for every input. This makes feature extraction slower (e.g., DINO gradients are ~15x slower to compute than just the forward pass). Additionally, the authors noted "diminishing returns" on models trained on gargantuan datasets (like EVA-CLIP, trained on 2 billion images), as these models are already so well-optimized that the SSL gradients provide less "new" information.
Future directions include exploring more efficient ways to compute these gradients and investigating if FUNGI can help detect "out-of-distribution" data or improve the reliability of Retrieval-Augmented Generation (RAG) systems.
Paper Citation: Walter Simoncini, Spyros Gidaris, Andrei Bursuc, Yuki M. Asano. "No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations." NeurIPS 2024. Code available on GitHub.
Summary updated