When engineers search for a neural network design for a new task, the most tempting shortcut is to reuse what worked on a related task. Sometimes that shortcut pays off. Often it does not: on TransNAS-Bench-101, a benchmark of seven vision tasks, copying the best architecture from one task to another sometimes does worse than random search. This proposal treats "should I transfer at all, and how much?" as the central question of cheap architecture search. We predict, before training anything on the target task, how well architecture rankings from a source task will carry over. We then pull the target task's cheap scoring model toward the source ranking only when that prediction clears a threshold. That threshold should rise as more trained target architectures become available.
The rule is borrowed from theory. Phase Transitions in Transfer Learning for High-Dimensional Perceptrons proves that in a simplified model, the optimal amount of transfer is exactly zero until source–target correlation crosses a threshold, and then it jumps. It also shows that soft transfer, which pulls the solution toward the source, beats hard transfer, which freezes it, and that more target data raises the threshold. Nobody has tested whether architecture search behaves this way. If it does, the theory gives a concrete recipe for the regime where search cost matters most, roughly zero to fifty trained target architectures.
Core Claim
Architecture search needs a way to estimate how much two tasks agree on which architectures are good before any target architecture is trained. We build that estimate from "zero-cost proxies": scores computed on an untrained network from a single minibatch, such as gradient norms or feature-rank measures. Our claim has two parts.
First, a transfer estimate built from zero-cost proxy scores predicts the true cross-task rank agreement between tasks. Its inputs are the proxy score vectors of a small fixed set of probe architectures, evaluated on each task at initialization, together with an initialization-time Fisher embedding in the style of AutoTransfer. The quantity it predicts is the Kendall correlation between true trained accuracies across the search space.
Second, a soft-transfer rule gated by this estimate beats both "always transfer" and "never transfer" at small labeled budgets. The gate uses a threshold that depends on the labeled budget, and we expect its empirically optimal value to increase with the number of trained target architectures, as the perceptron theory predicts.
The novelty is not the task fingerprint. NAS-Bench-Suite-Zero already observed that multi-proxy signatures track task similarity. It left closed-loop task-adaptive search open. The novelty is turning similarity into a calibrated, budget-dependent decision about whether to transfer, and testing whether that decision has the discontinuous shape the theory predicts.
Relation to Prior Work
Three existing systems move architectural knowledge across tasks, and each assumes transfer is worth doing once a neighbour is found.
Fisher Task Distance and its Application in Neural Architecture Search and Neural Architecture Search From Fréchet Task Distance measure an asymmetric Fisher-based distance to a bank of solved tasks. They then hard-prune the search space around the nearest task's best cell. That is the hard transfer the theory says is dominated, and it never declines to transfer.
Arch-Graph conditions a pairwise architecture comparator on a task embedding taken from a pretrained ResNet-50. It is the strongest reported method on our benchmark. It too has no mechanism for detecting a harmful source.
AutoTransfer supplies the method we reuse: Fisher condition numbers of randomly initialized anchor models, computed in seconds. It applies them only to warm-start graph neural network hyperparameter distributions, and it reports a modest Kendall correlation of 0.43 between embedding distance and performance rankings. We treat that modest number as a warning. Similarity alone is a weak signal, which is why we calibrate it into a transfer estimate rather than using nearest-neighbour retrieval.
Two results constrain the design. TransNAS-Bench-101 shows that direct transfer can underperform random search, and that architecture rankings across tasks are non-intuitive. For example, scene classification aligns with semantic segmentation more than with object classification. NAS-Bench-Suite-Zero shows that individual proxies drift across tasks and are biased toward model size. So our fingerprint uses many proxies plus FLOPs as an explicit covariate rather than trusting any single proxy.
Research Program
The target-side scoring model is a small regressor $f$ over zero-cost proxy features of an architecture $a$ evaluated on target data. It is fit to $n$ trained target architectures with accuracies $y_i$, and additionally regularized toward a source ranking $g_S$ over a pool $P$ of candidate architectures:
$$ \min_f \sum_{i=1}^{n} \big(f(a_i)-y_i\big)^2 + \lambda(\hat\rho, n) \sum_{a \in P} \big(f(a) - g_S(a)\big)^2 . $$
Here $g_S(a)$ is the standardized source-task accuracy of $a$, taken from the tabular benchmark. $\hat\rho$ is the predicted cross-task Kendall correlation, and $\lambda$ is the transfer strength. This is the architecture-search analogue of the $\ell_2$-toward-source soft transfer analysed in the perceptron paper. At $n=0$ the rule reduces to either the source ranking or the pure proxy ranking. The gate takes the form $\lambda = \lambda_0 \cdot \mathbb{1}[\hat\rho > \tau(n)]$, and we compare it against a smooth alternative that grows linearly in $\hat\rho$. Whether the step or the smooth version works better is itself an empirical test of the phase-transition picture. When several sources clear the gate, their rankings are averaged with weights proportional to $\hat\rho$.
The transfer estimator $\hat\rho$ must stay tiny, because there are few task pairs. It is a monotone, two-to-three-parameter map from a small set of inputs. One input is the Kendall correlation between proxy score vectors on roughly 100 probe architectures. Another is the distance between Fisher condition-number embeddings. The last is a size-bias control: the correlation of each proxy with FLOPs.
The training pairs come from all ordered task pairs within a shared search space. That gives 42 per TransNAS-Bench-101 space, plus the within-space pairs in NAS-Bench-Suite-Zero, such as the three NAS-Bench-201 datasets. The estimator is always evaluated with the target task held out.
The threshold $\tau(n)$ is fit per budget on held-out targets. The scientific question is whether it rises with $n$. A flat or falling threshold would falsify the transfer of the phase-transition result to architecture search, and it would be a publishable negative finding.
Baselines
The naive reference point is random search over the TransNAS-Bench-101 macro and micro spaces at matched sample budgets of 10, 25 and 50 trained architectures. Alongside it we run the always-transfer extreme: take the best architecture from the nearest source task, found either by human grouping or by fingerprint. TransNAS-Bench-101 reports that this sometimes loses to random search, so we must reproduce that failure.
The strongest existing method on this exact question is Arch-Graph. It reports reaching the top 0.16% of the macro space and the top 0.29% of the micro space with 50 sampled models, and an average rank of 5.24 on the macro space against 15.43 for weakNAS-t. We compare under the same leave-one-task-out protocol and the same 50-sample budget, and report the same statistics. Beating it means a lower average rank at 50 samples, or matching its top-percentile result with fewer samples. Arch-Graph also needs a pretrained ResNet-50 forward and backward pass on labeled target data to build its task embedding, while our fingerprint uses only untrained networks. We therefore also report wall-clock cost for embedding construction.
Two other baselines matter. REA-transfer, evolutionary search seeded from source populations, is reported by TransNAS-Bench-101 as finding 99th-percentile architectures across tasks. That serves as a secondary bar. Standalone AZ-NAS is the strongest pure zero-cost proxy. It reports Kendall 0.710 on NAS-Bench-201 ImageNet16-120, but we found no reported result on Lacuna for AZ-NAS on TransNAS-Bench-101, so we run it ourselves with its published settings.
The most dangerous baseline is the NAS-Bench-Suite-Zero surrogate: gradient-boosted trees on all 13 proxies plus FLOPs, trained on the same $n$ target architectures, reported to raise rank correlation by up to 42%. It uses exactly our feature set without any transfer, so the gap between it and our method isolates what transfer adds. Every learned method gets identical proxy features, identical target samples, and the same hyperparameter tuning budget. That budget is a fixed small grid chosen on non-target tasks only.
Three ablations isolate the key ingredient. Never-transfer sets $\lambda=0$ and is the surrogate above. Always-transfer uses a fixed $\lambda$ with no gate. Oracle-gate uses the true $\rho$, and shows how much is lost to estimation error.
Why not just…?
Why not just fit the multi-proxy surrogate on target samples and skip transfer entirely? This is the real threat. Our own model predicts that it should win or tie once $n$ is large, since the optimal transfer strength then goes to zero. We therefore claim gains only at small budgets, and the never-transfer ablation measures exactly where the curves cross. If the gated method fails to beat the surrogate at $n \le 25$ on at least the dense tasks, the proposal's premise is wrong. Those tasks are room layout, surface normals and segmentation, where TransNAS-Bench-101 shows the macro skeleton dominates and proxies calibrated on classification are least trustworthy.
Why not just transfer from the most similar task, as Fisher Task Distance does? Because similarity-ranked nearest neighbours can still sit below the useful threshold. AutoTransfer's 0.43 correlation between embedding distance and performance rankings shows how loose that link is. The always-transfer ablation, with nearest source chosen by our fingerprint, tests this directly. The pairs where it loses to random search are the cases the gate must catch.
Why not just use a strong zero-cost proxy like AZ-NAS or Dextr on the target? These scores are task-blind in structure. Dextr uses no labels at all, so it gives an architecture the same score whether the task is classification or surface normal estimation. Yet TransNAS-Bench-101 shows that rankings differ across tasks. We run both as baselines. A proxy that already matches Arch-Graph on dense tasks would make transfer unnecessary, and we would report that.
Why not just use Arch-Graph? It is the bar to beat, but it cannot decline to transfer, and it relies on a pretrained backbone. If our gate never fires differently from "always transfer," Arch-Graph's learned comparator probably dominates us, and the experiments will show that plainly.
Proposed Experiments
The first decisive check tests the transfer estimator. Across held-out target tasks in TransNAS-Bench-101 macro and micro, and the NAS-Bench-201 and NAS-Bench-301 tasks in NAS-Bench-Suite-Zero, we ask whether $\hat\rho$ rank-orders source–target pairs by true cross-task Kendall correlation. The Fisher-embedding-only estimator serves as the comparison, which tests whether proxy fingerprints add information beyond the AutoTransfer signal.
The second check is the main search result. We report average rank and top-percentile of the returned architecture at budgets 0, 10, 25 and 50 for the gated method against every baseline and ablation, with multiple seeds and paired comparisons per target task.
The third check tests the theory. For each budget we find the empirical break-even correlation, below which always-transfer loses to never-transfer, and test whether it increases with $n$. We also test whether a step gate beats a smooth one.
A final cost audit reports wall-clock time for the fingerprint and for Arch-Graph's embedding, following the warning in Zero-Cost Proxies for Lightweight NAS that savings in floating-point operations do not become wall-clock savings in proportion.
The proposal succeeds if the gated method beats the multi-proxy surrogate at small budgets and matches or beats Arch-Graph's reported macro average rank of 5.24 at 50 samples. It also needs the estimator to predict transfer agreement on held-out targets. A rising threshold would add independent support for the theory.
References
- Phase Transitions in Transfer Learning for High-Dimensional Perceptrons
- AutoTransfer: AutoML with Knowledge Transfer - An Application to Graph Neural Networks
- NAS-Bench-Suite-Zero: Accelerating Research on Zero Cost Proxies
- Fisher Task Distance and its Application in Neural Architecture Search
- Neural Architecture Search From Fréchet Task Distance
- Arch-Graph: Acyclic Architecture Relation Predictor for Task-Transferable Neural Architecture Search
- TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search
- AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search
- Dextr: Zero-Shot Neural Architecture Search with Singular Value Decomposition and Extrinsic Curvature
- Zero-Cost Proxies for Lightweight NAS