Large pretrained vision models such as DINOv2, CLIP, SAM and SigLIP are increasingly used frozen. A small trainable "probe" reads their features and solves a new task, which avoids fine-tuning. For classification, two findings stand out. Intermediate layers carry useful structure that the final layer throws away. Mixing several frozen models beats any one of them, and even beats a model distilled from all of them. Dense prediction, which labels every pixel as in segmentation, has been left behind. The method that picks layers adaptively averages away the spatial layout. The method that keeps the layout mixes layers and models with one fixed projection that is the same at every pixel.
We propose a probe that, at every image location, decides which (model, layer) features to trust. It treats the stack of features at a pixel the way cross-modal attention treats a second modality. A per-location query attends over a few dozen candidate features, and a learned gate falls back to a static mixture when the attention is not helping. The bet is specific and falsifiable: the best source varies within an image. Object boundaries and small parts should prefer shallow, high-resolution-sensitive layers, while object interiors and large stuff regions should prefer deep semantic layers. A single task-level weighting therefore leaves accuracy on the table. If an oracle analysis shows that per-region source choice has little headroom over the best fixed mixture, the proposal is wrong, and we test that first.
Core Claim and Its Place in Prior Work
Three groups of prior work frame the claim.
The first group motivates it by showing that static fusion fails in a spatially varying way. Static addition dilutes high-resolution detail when it is summed into coarse layers (Towards Region-Adaptive Feature Disentanglement and Enhancement for Small Object Detection). Fixed multi-scale aggregation cannot adjust to targets of different sizes (Context Contrasted Feature and Gated Multi-Scale Aggregation for Scene Segmentation). A single fixed blend of feature levels is suboptimal across diverse scenes (MoE-SPNet). All three findings come from networks trained end-to-end. None of them asks whether the same holds for the very different feature stacks of frozen foundation models.
The second group supplies the starting points we must beat.
Attentive Multi-Layer Fusion for Vision Transformers (ALF) uses one learned query to attend over a class token and an average-pooled token from every layer of a frozen ViT. It reports an average gain of 5.54 points over linear probing. Because it pools spatially, it cannot do dense prediction, and its layer choice is made once per image rather than per location.
Fantastic Features and Where to Find Them (ComBo) keeps patch grids. It resizes multi-layer maps from four frozen models to one grid, compresses them with a shared affine projection under an ℓ2 sparsity penalty, and runs a light transformer on the result. It reaches 78.2% on VTAB-1k against 77.5% for the distilled RADIOv2.5. Its mixing weights are the same at every pixel and every image, and the authors list dense tasks as future work.
Our probe sits between the two. It keeps ComBo's spatial grid and replaces its fixed projection with ALF-style attention, applied per location along the source axis rather than once per image.
The third group supplies mechanisms and design constraints.
An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention shows two things. Un-gated cross-attention lets a high-entropy stream wash out a weaker one. A gate of the form $g \odot \text{context} + (1-g) \odot \text{original}$, combined with sparsemax, preserves the identity of each stream. Here the "dominant modality" is a backbone with larger feature norms, and we reuse the gate in that form.
GateFusion applies the same principle hierarchically between audio and video encoders. It concerns 1D temporal signals rather than dense spatial maps.
Attentional Feature Fusion identifies the "initial integration bottleneck": a fusion weight computed from a naive sum of the inputs is computed from already-contaminated features. Its fix is to iterate the fusion once (iAFF). We borrow that two-pass idea for building the query.
Feature Pyramid Transformer reports that dot-product attention matches features poorly across scales and that negative Euclidean distance works better. That finding tells us the scoring function must be an explicit design variable rather than a default.
Research Program
All backbones stay frozen. For an image, we extract patch-feature maps from $L$ chosen layers of each of $M$ backbones and resize every map to a common grid at 1/16 of the input resolution. We use four models and four layers each, following ComBo's backbone set, which gives $S = M \cdot L = 16$ sources.
At grid location $p$, source $s$ contributes a token $z_{p,s}$. Each token passes through its own LayerNorm and a linear map to width $d$. The per-source normalization is deliberate: it removes the trivial "large norm wins" form of dominance before attention runs. Any remaining benefit of attention is then about content, not scale.
The fused feature at location $p$ is
$$ h_p = g_p \odot \sum_{s=1}^{S} a_{p,s}\, z_{p,s} + (1-g_p) \odot \bar{z}_p , $$
where the three terms are as follows.
The term $\bar{z}_p$ is a static mixture: a learned fixed weighting of the $z_{p,s}$, the same at all pixels, which is essentially ComBo's projection.
The weights $a_{p,s}$ are attention weights. They come from a per-location query $q_p$ scored against keys from each $z_{p,s}$, using softmax over dot products, sparsemax, or negative Euclidean distance. That choice is ablated.
The gate $g_p \in [0,1]^d$ comes from a 1×1 projection of $q_p$ and $\bar{z}_p$.
The query is built in two passes to address the initial integration bottleneck. The first pass uses a learned global query to produce a provisional fusion. The second-pass query $q_p$ is a projection of that provisional fusion plus a 3×3 neighborhood of it, so the selection at a pixel can see local context.
The gate guarantees that the probe can fall back to the static baseline. Attention is computed only along the source axis, so the cost is $O(HW \cdot S)$ rather than quadratic in pixels. This avoids the compute problem that forced FPT into locality constraints.
The fused map $h$ feeds a fixed light dense head shared by every method we compare: a two-layer transformer at 1/16 resolution, followed by convolutional upsampling.
Besides accuracy, the program produces an analysis artifact: per-pixel source-selection maps. We will measure whether selection shifts toward shallow layers as a function of distance to the nearest ground-truth boundary and of the size of the segment a pixel belongs to. We will also measure whether particular backbones dominate particular classes, for example SAM features at edges and CLIP features in semantic interiors. These maps are what would turn an accuracy number into an explanation.
Baselines
The naive reference point is a linear per-pixel probe on the final layer of the single best backbone. We will also run the same dense head on that single final-layer feature. Together these establish how much any multi-source fusion buys.
The static-learnable baseline normalizes every source, concatenates them, and applies a 1×1 convolution. This is learnable but input-independent channel weighting, and it is the most direct test of whether input dependence matters at all.
The strongest existing competitor is ComBo adapted to dense output. We keep its bilinear grid alignment, shared affine projection and ℓ2 backbone-sparsity penalty, and replace its classification readout with our shared dense head. It is the strongest because it already keeps spatial structure and combines both multiple models and multiple layers, which is exactly what our probe adds selection on top of. Its reported result is 78.2% global average accuracy on VTAB-1k (Fantastic Features and Where to Find Them). No reported result was found on Lacuna for ComBo, or for any frozen multi-backbone probe, on Cityscapes, LIP or COCO segmentation, so the dense numbers will be our own runs.
We also run a dense version of ALF. It learns one softmax weighting over the sources per image, computed from pooled tokens as in Attentive Multi-Layer Fusion for Vision Transformers, and applies that weighting uniformly across the grid. This isolates per-image from per-pixel selection.
Every method uses the same frozen features, the same head, the same data splits and the same optimizer. Each gets an identical hyperparameter search budget of 20 trials on learning rate and weight decay, and each is run with three seeds. The probes will also be parameter-matched within about 10% by widening the static baselines' projections.
Ablations isolate the key ingredient: the gate removed, the single-pass versus the two-pass query, the three scoring functions, and per-source normalization alone with no attention.
We count it as beating ComBo-dense if three conditions hold. The mean-IoU gain on Cityscapes and LIP must exceed the seed spread of both methods. Dense ALF must not close that gap. And the gain must concentrate in the boundary band and in small segments, as the claim predicts. A uniform small gain would suggest extra capacity rather than selection.
Why not just…?
Why not just use ComBo with a dense head? ComBo is the real threat. Its fixed projection followed by a transformer may already learn spatially varying behavior implicitly, because the transformer sees context. Nothing in the file shows that ComBo fails on dense tasks; it simply was not tested on them. Our comparison runs ComBo-dense with an identical head and budget. The boundary-stratified IoU and the oracle analysis below decide whether explicit per-pixel selection adds anything. If ComBo-dense matches us, we will report that as the finding.
Why not just normalize each source and concatenate? Much of the "modality dominance" reported for un-gated attention (Gated Cross Attention for DTI) may be a scale mismatch that normalization alone fixes. That is why the per-source-normalization-only ablation and the static concatenation baseline share our normalization. If they match the full probe, input-dependent weighting is unnecessary for frozen features. The evidence that concatenation overfits multi-layer stacks comes from classification with few labels (ALF) and may not transfer to dense tasks with many labeled pixels.
Why not just learn one weighting per task, as ALF does? If the best layer depends on the task but not on the region, per-task weights capture everything at a fraction of the complexity. The dense-ALF baseline and the oracle headroom check test exactly this. Regional failures of static fusion in trained networks (Bi et al., Ding et al.) make us expect region dependence, but that is a hypothesis, not evidence for frozen models.
Why not just fine-tune one backbone? ComBo reports about 3.2 GB of VRAM to adapt four backbones against over 21 GB for fine-tuning, and it beats a distilled student (ComBo). ALF reports roughly 36× faster training than fine-tuning. We will include one fine-tuned DINOv2 segmentation run as a ceiling reference, not as a competitor on the same budget.
Proposed Experiments and Decisive Checks
The first experiment is the headroom check, run before any probe is trained. On Cityscapes and LIP, we train one dense probe per source and evaluate an oracle that picks, for each ground-truth segment, the best source by that segment's IoU. We then compare this oracle against the best fixed mixture, using the static concatenation baseline. If the oracle's advantage is negligible, especially in boundary bands and small segments, we stop and report that frozen-feature selection is region-invariant. LIP's small, fine body parts make it the dataset most likely to show headroom. Cityscapes mixes thin structures with large stuff regions.
The main experiment compares all methods on Cityscapes and LIP mean IoU, with secondary evaluation on MS-COCO 2017. For COCO we use a frozen-feature mask head and report mask AP, following the dense suite in Feature Pyramid Transformer. That paper reports only relative gains (for example +1.6 mIoU on Cityscapes for its trained-backbone setting), so those figures calibrate expectations but are not directly comparable.
We stratify IoU by distance to boundary and by segment size, and correlate the selection maps with these quantities. As a guard against regressing on classification, we pool $h$ and evaluate on VTAB-1k against ComBo's 78.2%.
The scoring-function ablation tests whether FPT's finding that negative Euclidean distance beats dot products carries over from cross-scale matching to cross-model matching. The gate ablation tests whether, without the gate, attention collapses onto one backbone; we diagnose collapse by the entropy of $a_{p,s}$ and by the backbone share in the selection maps.
We will treat the proposal as confirmed only if all three of the following hold. The oracle shows region-dependent headroom. The probe recovers a meaningful fraction of it over ComBo-dense and dense ALF. And its selection maps move toward shallow layers near boundaries.
References
- Towards Region-Adaptive Feature Disentanglement and Enhancement for Small Object Detection
- Context Contrasted Feature and Gated Multi-Scale Aggregation for Scene Segmentation
- MoE-SPNet: A mixture-of-experts scene parsing network
- Attentive Multi-Layer Fusion for Vision Transformers
- Fantastic Features and Where to Find Them: A Probing Method to combine Features from Multiple Foundation Models
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- GateFusion: Hierarchical Gated Cross-Modal Fusion for Active Speaker Detection
- Attentional Feature Fusion
- Feature Pyramid Transformer