Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models

Benjamin Ramtoula, Pierre-Yves Lajoie, Paul Newman, Daniele De Martini

Introduction

The current landscape of computer vision is dominated by "Foundation Models" (FMs) trained with diverse objectives: CLIP focuses on text-image alignment, MAE on reconstruction, and SAM on segmentation. These varying objectives lead to complementary strengths; for instance, one model might excel at semantic object identification while another better understands geometric spatial relationships.

Ideally, researchers would combine these models to leverage their collective knowledge. However, existing integration methods face significant hurdles. Full fine-tuning or Parameter-Efficient Fine-Tuning (PEFT) of multiple massive models is computationally prohibitive. Model distillation, while effective, is slow and often results in the loss of unique features from the teacher models. Traditional linear probing is efficient but often relies on global average pooling, which destroys the spatial information necessary for structured tasks like distance estimation or object counting.

The ComBo Architecture

To address these challenges, we introduce Combined backBones (ComBo), a lightweight adapter that treats multiple frozen FMs as a unified feature library. ComBo avoids backpropagation through the massive backbone models, making it highly efficient.

The ComBo architecture extracts features from multiple frozen foundation models, processes them through a lightweight transformer, and feeds the result to a task-specific head.

The ComBo pipeline consists of four key stages:

  1. Feature Extraction & Alignment: Feature maps are extracted from multiple layers across several models. To handle varying token counts (e.g., between DINOv2 and SigLIP), bilinear interpolation aligns all features to a consistent spatial grid.
  2. Layer Embedding Compression: Instead of using a massive linear layer, ComBo employs a small affine projection shared across all spatial positions. This compresses the high-dimensional feature stack into compact "multi-model tokens."
  3. Transformer Processing: These tokens are processed by a lightweight transformer encoder, allowing the model to dynamically attend to the most relevant features across different spatial locations and backbones.
  4. Task-Relevance Mechanism: A sparsity-inducing regularization term ($\ell_2$ norm on projection weights) is added to identify the most useful backbones for a given task, allowing for efficient model pruning.

Key Insights: Spatial Information and Intermediate Layers

A critical finding of this work is that spatial information is paramount for "Structured" tasks. By preserving patch-wise structure through a transformer-based approach rather than aggressive pooling, ComBo achieves a +22% improvement over standard linear probing on structured benchmarks.

Furthermore, the "gold" features for a task are often found in intermediate layers rather than the final output. As shown in the validation accuracy plots below, different models peak at different depths depending on the task. ComBo naturally identifies and utilizes these optimal features through its learned projection mechanism.

Validation accuracy across different layers for SVHN and Clevr/distance tasks. Different foundation models show peak performance at varying intermediate layers depending on the task requirements.

Experimental Results

We evaluated ComBo on the VTAB-1k benchmark, which spans 19 diverse tasks categorized into Natural, Specialized, and Structured domains.

  • State-of-the-Art Probing: When probing a single ImageNet-21K ViT, ComBo achieved 74.6% average accuracy, surpassing prior leaders like SMP (73.6%) and Head2Toe (70.9%).
  • Multi-Model Superiority: Combining four backbones (DFN CLIP, DINOv2, SAM, and SigLIP) resulted in a 78.2% global average accuracy. This outperformed RADIOv2.5 (77.5%), a state-of-the-art model explicitly distilled from the same backbones, suggesting that direct probing can access features that distillation might lose.
  • Efficiency: ComBo is remarkably resource-efficient, requiring only ~3.2 GB of VRAM to adapt four backbones—compared to over 21 GB for standard fine-tuning. Training can be completed in approximately 10 minutes on a single consumer GPU.

Conclusion and Future Work

ComBo demonstrates that sophisticated feature merging does not require massive compute. By intelligently probing frozen models, we can achieve performance that rivals or exceeds expensive distillation and fine-tuning methods. Future research will focus on extending ComBo to dense prediction tasks like semantic segmentation and automating the selection of the optimal number of backbones for any given task.

Summary updated

BibTeX
@inproceedings{ramtoula2025fantastic,
  author = {Benjamin Ramtoula and Pierre-Yves Lajoie and Paul Newman and Daniele De Martini},
  title = {{Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=PxgIElCohI},
}
Figure