Much of computer vision progress is reported as a new "head": a specialized module bolted onto a feature extractor (the backbone) to solve segmentation, few-shot matching, or language grounding. These heads are almost always introduced and compared on one older backbone, usually ResNet-50. Meanwhile, simply swapping in a modern pretrained transformer often produces large gains of its own. Nobody knows how much of a head's published advantage survives once the backbone is modern and every competing system gets the same tuning effort. This proposal measures exactly that. We train a ladder of five backbones crossed with published heads and with deliberately simple heads, under one equalized hyperparameter-search budget. The quantity we report is the head gain: how much the specialized head beats a simple head on the same backbone, and how that gain changes as the backbone improves.
The core hypothesis is that published heads split into two kinds. Compensatory heads repair something the old backbone lacked, such as spatial resolution or global context, and their gain should shrink toward zero on strong backbones. Structural heads encode something the task itself requires, such as one-to-one assignment of masks to objects, and their gain should persist. If the hypothesis holds, the field gets a cheap test to run before publishing a head: re-measure the gain on one modern backbone. If it fails and gains turn out to be additive across backbones, that is also useful, because it would justify the current habit of evaluating heads on ResNet-50 alone.
Core Claim
Let $s(h, b)$ be the validation-selected test score of head $h$ on backbone $b$, where each pair is tuned with the same number of automated search trials. Let $h_0$ be a minimal head for the same task. The head gain is
$$\Delta_h(b) = s(h, b) - s(h_0, b).$$
Prior work reports only $\Delta_h(\text{ResNet-50})$, and usually under asymmetric tuning. We claim three things. First, for compensatory heads $\Delta_h(b)$ falls with backbone quality, and the gain often drops inside seed noise on self-supervised transformer backbones. Second, structural heads keep most of their gain. Third, which group a head falls into can be predicted from a cheap measurement of what the backbone lacks for that task. "Additive" here means $\Delta_h(b)$ is roughly constant across $b$; we test this as a head-by-backbone interaction term in a two-way analysis over seeds.
Grounding in Prior Work
Two controlled re-evaluations supply the method, and both found that much reported progress came from training recipes rather than architecture. Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline standardized augmentation and validation-based model selection. Under that protocol, PointNet++ rose from 90.0% to 93.3% on ModelNet40, and a 0.8M-parameter 2D-projection baseline beat specialized 3D models. Disentangling Architecture and Training for Optical Flow applied one modern recipe to PWC-Net, IRR-PWC and RAFT. Older models gained up to 30%, but RAFT-it still led, with 4.31% Fl-all on KITTI 2015.
Both studies vary the architecture as a whole under a fixed recipe. Neither crosses the backbone with the head, which is the factorization this proposal needs. The optical-flow result also constrains us: architecture differences did not vanish entirely. So we predict some heads survive rather than claiming all gains are artifacts.
Three papers supply the case for equal tuning budgets rather than equal hyperparameters. On the State of the Art of Evaluation in Neural Language Models used automated black-box search with equal budgets and found a regularized LSTM reached 58.3 test perplexity on Penn Treebank, against 62.4 for NAS cells and 65.4 for Recurrent Highway Networks. Granero Moya et al. and Jiang et al. showed that uniform hyperparameters across architectures systematically favor some models over others. Uniform settings therefore cannot be our fairness device.
Two benchmarks are the closest standardized efforts on the backbone side, and both stop short of complex heads. ELEVATER auto-tunes adaptation of language-augmented backbones across 55 datasets, but only with classification projections and standard detectors. Synesis shows that backbones with equal probe accuracy can organize their features very differently, but it measures only with linear or shallow probes. Our prediction experiment asks whether such measurements predict head gain.
Simple Local Attentions Remain Competitive for Long-Context Tasks is a warning for that experiment. A synthetic proxy benchmark (Long Range Arena) clustered all models at 60–61% while their real-task scores diverged widely, so a geometry-based predictor could turn out to be equally hollow.
The heads come from four papers. Each was motivated by a backbone weakness, which makes each a candidate compensatory head:
- DUpsampling (Decoders Matter for Semantic Segmentation) is a learned upsampler. It exists so that CNNs can avoid expensive high-resolution dilated convolutions.
- Dynamic prototype convolution (DPCN) captures fine object detail for few-shot segmentation. Its authors note that it remains conditioned on ResNet-50 features.
- Locate Then Segment (LTS) uses an explicit position prior for referring segmentation, motivated by weak spatial anchoring in CNNs.
- UniInst (UniInst) uses one-to-one bipartite mask matching for NMS-free instance segmentation.
We pre-register UniInst's matching as the likely structural head. It addresses duplicate predictions, which arise from how a dense predictor assigns outputs to objects regardless of backbone, so better features should not remove the need for it.
Research Program
The backbone ladder has five rungs: ResNet-50 with supervised ImageNet pretraining, then supervised ViT-B, CLIP ViT-B, MAE ViT-B and DINOv2 ViT-B. The last four have similar size and differ mainly in pretraining objective. This means the ladder varies both architecture family and feature quality. Comparing the four ViT-B variants with each other separates the two. Transformer backbones get a standard multi-scale adapter shared across all heads, so no head receives private help.
Each of the four tasks gets the published head and a minimal head:
- PASCAL VOC 2012 segmentation. DUpsampling against bilinear upsampling with a one-layer classifier.
- PASCAL-5i one-shot segmentation. DPCN against cosine matching to a masked-average prototype.
- RefCOCO, RefCOCO+ and RefCOCOg. LTS against single-stage feature fusion with a light mask decoder.
- COCO and OCHuman instance segmentation. UniInst against CondInst with NMS. This pairing is chosen because the published OCHuman gap is the largest head effect in our sources.
Every head-backbone cell receives the same automated budget. We use Bayesian search over learning rate, backbone layer-wise decay, weight decay, warmup and augmentation strength, run on a shortened schedule. The best configuration is then trained at full length with three seeds. Model selection uses validation data only, following the test-leakage lesson of SimpleView. Before running the factorial, we reproduce each published head on ResNet-50 to within a stated tolerance of its paper. That check guards against blaming a head for our own implementation errors.
The prediction experiment pits three candidate predictors of $\Delta_h(b)$ against each other, all computed on the frozen backbone without training any head:
- global linear-probe accuracy;
- the Synesis-style invariance, equivariance and disentanglement scores;
- a task-matched deficit probe. For resolution-oriented heads this is a frozen dense linear probe's boundary-region IoU. For grounding heads it is a frozen probe's localization accuracy for the referred box.
We judge the predictors by rank correlation with measured head gain, using leave-one-head-out evaluation.
Baselines
Naive reference: simple head on each backbone. The reference point is the minimal head $h_0$ on every backbone, tuned with the same budget. This is the "do nothing clever" system, and our expectation is that its rows improve fastest as the backbone improves. If even this baseline cannot approach published head scores on DINOv2, the compensatory story is wrong for that task.
Strongest competitor: each published head at its reported operating point. These are the published heads as their papers report them:
- DPCN: 66.7% mIoU one-shot on PASCAL-5i with ResNet-50.
- UniInst: 39.0 mask AP on COCO with ResNet-50, and 40.2 with ResNet-101. Its reported margin over CondInst on OCHuman is +12.6 mask AP.
- DUpsampling: 88.1% mIoU on PASCAL VOC 2012 and 52.5% on PASCAL Context.
- LTS: +3.2 IoU on RefCOCO+ and +3.4 on RefCOCOg over prior methods. We found no reported absolute LTS number on RefCOCO in our sources.
We rerun these heads in our harness with the same data, schedule length and search budget as everything else. "Beating" a head means one of two things. Either the simple head on a modern backbone matches or exceeds the specialized head on that backbone, with the confidence interval of $\Delta_h$ over seeds containing zero or lying below it. Or the specialized head's gain on the strongest backbone is less than half its ResNet-50 gain.
Ablation 1: asymmetric tuning. This reproduces common practice by giving the published head the full search budget and the simple head the defaults from the original papers. The difference between this condition and the equal-budget condition tells us how much of each published gain tuning asymmetry creates.
Ablation 2: uniform hyperparameters. All cells share one configuration. This ablation tests the claim from Granero Moya et al. in the vision setting.
Ablation 3: same family, different objectives. Comparing results across the four ViT-B variants isolates feature quality from architecture family.
Why not just…?
Why not just attach a linear head to a foundation backbone and declare complex heads obsolete? This is the most serious threat, and the proposal is built to test it, not to assume it. ELEVATER shows strong linear probing in few-shot classification. SimpleView shows simple designs winning under fair training. But both cover classification only. On dense tasks, the OCHuman gap and the LTS gains are evidence that structure can matter. Our "simple head on each backbone" rows answer the question directly, task by task, and we expect a split verdict rather than a uniform one.
Why not just modernize the training recipe of old models, as the optical-flow study did? A recipe change alone cannot tell a head's effect apart from a backbone's effect, because both change together. Even in the optical-flow study, RAFT kept its lead after the recipe update. Ablation 1 measures the recipe-and-tuning share of each gain. The factorial measures the share that remains.
Why not just fix identical hyperparameters for everyone? Prior evidence says this biases results toward whichever model suits the chosen settings (Jiang et al.). Ablation 2 shows whether that bias actually changes the head ranking here.
Why not just rank backbones with probes or geometry scores and skip training heads? The LRA result shows that proxy benchmarks can be uninformative. The prediction experiment tests this possibility explicitly. If linear-probe accuracy predicts head gain as well as the deficit probe does, we will report that and drop the geometry claim.
Decisive Checks
The study's central result is the interaction test. Suppose the published-head gains are additive, meaning $\Delta_h$ is flat across the ladder, and this holds for DUpsampling and DPCN, our strongest compensatory candidates. Then the core claim is refuted. A second outcome would also count against our framing: if UniInst's OCHuman gain collapses on DINOv2 while DUpsampling's gain persists, the compensatory-versus-structural split as pre-registered is wrong.
A positive result has three parts:
- the gains of DUpsampling, DPCN and LTS shrink by more than half from ResNet-50 to DINOv2 under equal budgets;
- UniInst retains most of its OCHuman advantage;
- the deficit probe ranks head gains better than global probe accuracy does.
If all three hold, the field would gain a concrete rule: a new head should be shown to keep its gain on a modern backbone under equal tuning before it counts as progress.
References
- Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline
- Disentangling Architecture and Training for Optical Flow
- On the State of the Art of Evaluation in Neural Language Models
- A Comparative Analysis of Pretrained Language Models for Text-to-Speech
- LV-UNet: A Lightweight and Vanilla Model for Medical Image Segmentation
- ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models
- Position Paper: Towards a Unified Representation Evaluation Framework Beyond Downstream Tasks
- Simple Local Attentions Remain Competitive for Long-Context Tasks
- Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation
- Dynamic Prototype Convolution Network for Few-Shot Semantic Segmentation
- Locate Then Segment: A Strong Pipeline for Referring Image Segmentation
- UniInst: Unique representation for end-to-end instance segmentation