A pretrained image network produces useful features at every depth. Early layers encode edges and textures, and deep layers encode object identity. Prior work shows that fusing several depths helps, both for dense tasks and for robustness when the test images come from a new style or source. Fusing everything, though, yields a huge descriptor, and much of it is noise or source-specific style. We propose a training-free filter that sits in front of a frozen backbone. It scores every channel at every depth without class labels, using a classical graph-smoothness criterion. It keeps a channel only if that channel's score ranks it highly consistently: across each training domain separately, and across a sweep of the graph's own settings. A plain linear classifier is then trained on the surviving channels.
The idea in two sentences: an unsupervised feature score whose verdict flips when you change the source domain, or when you change the neighborhood size of the similarity graph, is probably reacting to style or to a kernel artifact rather than to content, so we keep only features on which these views agree. This removes the weighted multi-loss training that current approaches to fusing depths for robustness depend on. That training is the main source of their tuning brittleness. It also makes a well-known weakness of the graph criterion, its sensitivity to its own settings, into the selection signal itself.
Core Claim
We claim the following. A label-free, consensus-filtered subset of multi-depth channels, fed to a linear classifier on a frozen backbone, will beat both a final-layer linear classifier and a full-concatenation linear classifier on held-out domains. It will do so at a fraction of the concatenated dimension and under leakage-free model selection. The filter has no loss weights. Its out-of-domain accuracy should vary much less across its own settings than a learned sparse-reconstruction selector varies across its loss weights.
What would be new is the combination of three things. Selection is done across intermediate depths rather than on raw inputs or the last embedding. The criterion uses no class labels. Disagreement across domains and across graph settings is used as evidence of noise. That last element is the step borrowed from gradient-agreement methods in domain generalization, applied here to feature scores instead of gradient coordinates.
Where This Sits
Three bodies of work supply the pieces, and each leaves out what we add.
The first supplies the scoring tool. The Laplacian Score (Laplacian Score for Feature Selection) builds a nearest-neighbor graph with a heat-kernel weight of bandwidth $t$ over $k$ neighbors. It rewards features that vary smoothly over that graph, and under some conditions it coincides with the supervised Fisher Score. Its known weakness is that rankings change markedly with $(k, t)$ and offer no internal check. It has been applied to raw inputs, never to intermediate network channels or across domains. The AutoEncoder Feature Selector (AutoEncoder Inspired Unsupervised Feature Selection) learns a reconstruction network with a group-sparsity penalty on its input weights. It beats linear self-representation, for example 73.1% versus 60.5% clustering accuracy on Prostate_GE. However, it introduces a reconstruction-versus-sparsity weight, non-convex training, and in-distribution-only evaluation. We treat it as the strongest embedded competitor, not as our method.
The second body motivates the depth axis. Concatenating AlexNet Conv1 through Conv5 raised contour-detection ODS from .699 (Conv2 alone) to .741 on BSDS500, at the cost of a 1,376-dimensional per-pixel descriptor (Pixel-wise Deep Learning for Contour Detection). CLIPCEIL fuses all ViT layers through an adapter and refines channels by trading inter-domain against inter-class variance. Its authors argue that intermediate layers carry domain-invariant structure. $M^2$-CL (Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalization) attaches extraction blocks to 13 ResNet-18 layers with a class-contrastive loss. Both of these need class labels in the selection or fusion step, both statically weight several losses, and both pay heavy memory costs. We keep their premise that depth helps under shift, and drop the labels, the learned adapter, and the weighted objectives.
The third body supplies the agreement principle and the evaluation discipline. Gradient surgery (Domain Generalization via Gradient Surgery) zeroes parameter updates whose sign disagrees across source domains, treating disagreement as domain noise. It found this works better than projection-based conflict resolution (PCGrad) for domain generalization, at 65.80% versus 63.89% for ERM on PACS with AlexNet. Fish (Gradient Matching for Domain Generalization) aligns gradients across domains. We reuse the "agreement across domains means signal" logic at the level of label-free feature scores, which requires no backward passes at all. DARLING shows that domain metadata without class labels can drive robust representation learning, but it works only on the final layer and does no selection. In Search of Lost Domain Generalization constrains the whole project. Under training-domain validation, tuned ERM reached 66.6% average on DomainBed with ResNet-50, and no specialized method beat it by more than about one point. Any gain we report must clear that bar under the same protocol.
Research Program
The pipeline has no trained components except the final linear classifier.
First, we pass source-domain images through a frozen backbone. We use ResNet-50 for the DomainBed setting, and separately the CLIP image encoder used by CLIPCEIL. From every stage or block we take each channel, global-average-pool it to one number per image, and standardize it.
Second, within each source domain $d$ and for each setting $(k, t)$ in a fixed log-spaced grid $G$, we build a nearest-neighbor graph over that domain's images. We then compute every channel's Laplacian Score, convert the scores to a rank $r_c(d, k, t)$, and normalize it to $[0,1]$ so that 0 is best. The default graph is built on the PCA-whitened concatenated multi-depth descriptor. The final-layer embedding is the main ablation.
Third, we combine the ranks into one consensus score per channel $c$:
$$s_c = \operatorname{median}_{d,(k,t)} \, r_c(d,k,t) \; + \; \operatorname{IQR}_{d,(k,t)} \, r_c(d,k,t),$$
where the first term rewards channels that are smooth on average and the interquartile range penalizes channels whose verdict depends on which domain or which graph setting is used. Adding the two terms uses a single fixed unit weight, deliberately not tuned.
We keep the $m$ channels with the lowest $s_c$. As a redundancy guard, we drop any channel whose correlation with an already-kept channel exceeds 0.95. We choose $m$ by training-domain validation, the only selection knob, and we report results across all $m$.
Fourth, we train a linear classifier on the kept channels with class labels from the source domains, as ERM does.
Two analyses accompany the method. We report which depths survive selection, testing CLIPCEIL's claim that intermediate layers carry the invariant content, now without labels. We also measure ranking stability as Kendall's tau between top-$m$ sets under perturbed $(k, t)$, bootstrap resamples, and held-out source domains. This connects to the stability concerns in Algorithmic stability and generalization of an unsupervised feature selection algorithm.
For single-source generalization, where real domains are absent, we will try augmentation-defined pseudo-domains: color, texture, and blur families. This is a secondary test, and we expect it to be weaker.
Baselines
The naive reference is a linear classifier on the frozen backbone's final-layer pooled features, trained with ERM on pooled source data. It tells us whether multi-depth features help at all under a frozen backbone.
The natural fusion reference is full concatenation: the same linear classifier on all pooled channels from all depths, with L2 regularization strength tuned by training-domain validation. This is the strongest simple competitor, because a tuned regularized linear classifier may down-weight noisy channels on its own. Our method must beat it on held-out domains, not just match it at lower dimension, for the domain-consensus claim to stand.
The strongest end-to-end reference on ResNet-50 is fine-tuned ERM as reported in In Search of Lost Domain Generalization: 66.6% average accuracy across DomainBed under training-domain validation, 77.5% on VLCS, and 66.5% on Office-Home. CORAL is the best specialized method there at 67.5%. A frozen-backbone method that falls within a point of fine-tuned ERM while training only a linear layer is already an interesting result. Exceeding 67.5% would count as beating the strongest DomainBed method. On the CLIP backbone, the strongest reported method is CLIPCEIL at 77.3% average across its benchmarks, against 70.2% for zero-shot CLIP and 75.0% for the previous best. CLIPCEIL uses class labels during selection, so landing between 75.0% and 77.3% without labels in the selection step would be informative. Exceeding 77.3% would be a clear win. We will rerun CLIPCEIL with released or reimplemented code on our splits rather than trust cross-paper comparisons.
The strongest unsupervised selection competitor is AEFS applied to the concatenated multi-depth descriptor. It is trained per method with the same compute, and its reconstruction-versus-sparsity weight is swept over the same number of trials we give our $m$ and grid choices.
The ablations that isolate the key ingredient are the following. We drop the IQR term, which leaves a pooled Laplacian Score over all domains with no consensus. We use a single $(k, t)$ instead of the grid. We use random channels at matched $m$ and matched per-depth proportions. Finally, we use a label-free invariance-only filter that ranks channels by the ratio of between-domain to within-domain variance, ignoring smoothness.
All methods share the backbone, the DomainBed splits, training-domain validation, the same number of random-search trials, and three seeds.
Why not just…?
Why not just concatenate all depths and let a regularized linear classifier sort it out? This is the most serious threat, because L2 or L1 regularization on the classifier is a form of selection. The difference is that the classifier selects by source-domain label fit. Gulrajani and Lopez-Paz show that source fit tends to reward whatever predicts well in the training domains, including style. Our filter selects by agreement across domains before any labels are seen. The full-concatenation baseline settles this directly. If it matches us on held-out domains at every $m$, the consensus idea adds nothing beyond compression, and we will report that.
Why not just filter channels by how much their means differ across domains, which is simpler and also label-free? An invariance-only filter can keep channels that are constant everywhere and therefore useless, and it cannot tell content from noise within a domain. Smoothness over the image-similarity graph supplies that missing content signal, which is what Laplacian Score was designed for. We include this filter as an ablation. If it does as well as the full consensus score, we will adopt it, because a simpler method that wins should replace ours.
Why not just fine-tune with ERM, since tuned ERM matches most specialized methods? Fine-tuned ERM is the bar, and we do not expect a frozen linear classifier to beat it everywhere. Our claim is narrower. The method trains only a linear layer, needs no loss weights, and should help most where depth matters and labels are scarce. DARLING's 1%-label DomainNet setting, where self-supervised baselines reach only 10.05% to 11.80% and DARLING reaches 13.29%, is where a fixed, label-free selection should help most. We report the comparison in both regimes.
Why not just learn the selection with a sparse autoencoder on the fused features? That is AEFS moved to depth. It brings back exactly the reconstruction-versus-sparsity weight whose brittleness is documented for composite losses (Generating Images with Perceptual Similarity Metrics based on Deep Networks, Domain Separation Networks). It is run as a baseline with an equal tuning budget, and we report the spread of its held-out accuracy across its weight sweep next to ours.
Proposed Experiments
The main experiment is DomainBed: PACS, VLCS, Office-Home, TerraIncognita, and DomainNet, using leave-one-domain-out targets and training-domain validation only. It is run on frozen ResNet-50 and frozen CLIP features. A second experiment is WILDS Camelyon17, where the tissue-scale structure should reward intermediate depths. There ERM reports 70.3% and Fish 74.7% (Gradient Matching for Domain Generalization). Both numbers come from fine-tuned models, so we label the comparison accordingly. A third experiment is the 1%-label DomainNet regime from DARLING. As a dense-task efficiency check, we apply per-pixel channel selection to BSDS500 using a single domain and the $(k, t)$ consensus only, asking whether a small selected subset recovers the .741 ODS of full Conv1 through Conv5 concatenation.
Four checks decide the outcome. The first is whether the consensus-selected classifier beats the tuned full-concatenation classifier by more than one point in average held-out accuracy across seeds. One point is the margin DomainBed found that specialized methods could not reliably clear. The second is whether removing the IQR term, that is, removing the domain and setting consensus, erases most of that gain while random channels at matched $m$ do not reproduce it. The third is whether held-out accuracy varies across the $(k, t)$ grid and the choice of $m$ by clearly less than AEFS varies across its loss weight. That would show the sensitivity has been removed rather than relocated. The fourth is whether the kept channels concentrate in intermediate depths. If the first check fails, the paper reports a cheap compression method, not a robustness method. If the second fails, the robustness comes from depth alone and not from agreement.
References
- Laplacian Score for Feature Selection
- Autoencoder Inspired Unsupervised Feature Selection
- Pixel-wise Deep Learning for Contour Detection
- CLIPCEIL: Domain Generalization through CLIP via Channel rEfinement and Image-text aLignment
- Multi-Scale and Multi-Layer Contrastive Learning for Domain Generalization
- Domain Generalization via Gradient Surgery
- Gradient Matching for Domain Generalization
- Towards Unsupervised Domain Generalization
- In Search of Lost Domain Generalization
- Algorithmic stability and generalization of an unsupervised feature selection algorithm
- Generating Images with Perceptual Similarity Metrics based on Deep Networks
- Domain Separation Networks