TL;DR
FUNGI (Features from UNsupervised GradIents) is a "plug-and-play" method that boosts the performance of frozen, pretrained transformer models without any additional training. By concatenating standard model embeddings with projected gradients from self-supervised objectives (like DINO or SimCLR), the authors achieve significant gains in classification, retrieval, and scene understanding across vision, text, and audio domains.
What Problem Does This Paper Address?
Deep learning typically follows a "pretrain then fine-tune" paradigm. While pretrained backbones provide excellent general-purpose features, their frozen embedding spaces are often suboptimal for specific downstream tasks. Fine-tuning solves this but is computationally expensive and requires labeled data.
On the other hand, non-parametric methods like $k$-Nearest Neighbors ($k$NN) are efficient and require no training, but they are entirely dependent on the quality of the frozen representation. The authors ask: Can we extract more "adaptive" information from a frozen model for a specific data point without actually updating the weights?
Methods
The core idea of FUNGI is to treat the gradient of a self-supervised loss as a feature itself.
The process works as follows:
- Compute SSL Losses: For a given input (e.g., an image), the model calculates several self-supervised learning (SSL) losses. The paper uses three families: DINO (distillation/clustering), SimCLR (contrastive learning), and a KL-divergence loss (entropy maximization).
- Extract Gradients: Instead of updating weights, the method performs a backward pass to extract the per-sample gradients with respect to a specific layer (typically the last attention output projection).
- Dimensionality Reduction: Gradients are massive. To make them usable for retrieval, FUNGI uses binary random projections to squash these high-dimensional gradients down to the size of the original embedding.
- Feature Fusion: The original embedding and the projected gradients are concatenated and then further compressed using PCA to maintain a constant feature size (iso-storage).
This approach essentially captures "first-order learning signals"—how the model would change to adapt to that specific input—and uses that information to enrich the representation.
Datasets
The authors conducted an extensive evaluation across multiple modalities:
- Vision: 11 datasets including ImageNet-1K, CIFAR-10/100, Oxford Flowers 102, CUB-200, and Stanford Cars.
- Text: 5 datasets including TREC (question classification), Banking-77, and AG News.
- Audio: ESC-50 (environmental sounds) and SpeechCommands V2.
- Dense Prediction: Pascal VOC 2012 and ADE20K for semantic segmentation.
Key Insights and Ideas
- Gradients are Complementary: The "aha moment" is that gradients from different SSL objectives capture fundamentally different information than the standard embedding. As shown in the paper, these features have low similarity to each other (measured by CKA), meaning their combination provides a much richer description of the data.
- Diversity Wins: Combining multiple types of gradients (e.g., KL + DINO + SimCLR) consistently outperforms using just one.
- Deeper is Better: Gradients extracted from the final layers of a transformer are significantly more predictive than those from earlier layers, mirroring how semantic information is built up in the forward pass.
- Zero-Shot Adaptation: FUNGI acts as a form of non-parametric transfer learning. It allows a frozen model to "sense" the local curvature of the loss landscape for new data, providing a task-specific signal without a single weight update.
Key Results
FUNGI delivers consistent improvements across 20 different backbones (ViT, DeIT, CLIP, DINOv2, etc.):
- Semantic Segmentation: Using FUNGI features improved a frozen DINO model by a staggering +17% mIoU on Pascal VOC without any training.
- Classification: On average, FUNGI improved $k$NN accuracy by 4.4% on the full dataset and 2.8% in 5-shot scenarios across 11 vision benchmarks.
- Data Efficiency: The method is particularly potent in low-data regimes, showing significant boosts when only 3 to 6 examples per class are available.
- Multimodal Success: The benefits extended to BERT and T5 for text, and SSAST for audio, proving the method's generality.
Limitations and Future Work
- Computational Cost: The main drawback is the overhead. Computing gradients requires a backward pass, which is slower than a simple forward pass. For example, SimCLR gradients can be ~40x slower to compute than standard embeddings due to the multiple views required.
- Diminishing Returns with Scale: The authors noted that as the pretraining dataset size grows (e.g., moving from ImageNet-1K to 2B image-text pairs), the relative gain from FUNGI decreases, likely because the base embeddings are already highly optimized.
- Text Augmentation: The SimCLR objective for text relied on simple word deletion; the authors suggest that more sophisticated augmentations (like back-translation) might yield even better text gradients.
In summary, FUNGI proves that there is a wealth of untapped information hidden in the gradients of frozen models, offering a powerful new tool for researchers working with fixed feature extractors.
Summary updated