Neural architecture search spends most of its compute training candidate networks to find out how good they are. Cheap "zero-cost proxies" can score an untrained network in one forward and backward pass on a single minibatch, but no proxy is trustworthy everywhere. A proxy that ranks architectures well on one benchmark can rank them backwards on another. This proposal treats "how much should I trust each proxy on this new task" as a small-sample estimation problem, the kind hierarchical Bayesian models were built for. Each proxy's reliability on a new search task is modeled as a draw from a population of reliabilities seen on earlier tasks. Its estimate starts near that population and moves toward the task-specific value as fully trained evaluations accumulate.
The practical claim is that this shrinkage matters most in the first five to twenty full trainings of a search, which is also where a search spends its most consequential and most wasted compute. Existing methods estimate proxy trust either from scratch inside each run, which is too noisy with a handful of points, or by fixing one proxy recipe for all tasks, which fails when a proxy flips sign. Partial pooling sits between the two. It should pick good candidates earlier, measured in full training runs and in wall-clock time.
Core Claim
A hierarchical Bayesian model of proxy reliability, fit across tasks and updated on each new task, reaches a given search regret with fewer full training runs than within-run reliability tracking or a fixed cross-task proxy recipe. The model treats each proxy's slope and noise on each task as latent variables, with a population prior over tasks. The gain should be concentrated in the low-evaluation regime. As evaluations accumulate, the method should converge to per-task estimation, because shrinkage weakens once the task's own data are plentiful. The model is small: a few parameters per proxy per task, with the latent architecture accuracies integrated out in closed form. Its inference cost is seconds and does not reintroduce the cost it aims to remove.
Where This Sits
The motivating evidence comes from three places. Zero-Cost Proxies for Lightweight NAS introduced saliency proxies such as synflow, snip, grasp, fisher and jacob_cov, and showed a voting ensemble reaching Spearman 0.82 on NAS-Bench-201. It also documented that theoretical proxy savings shrink on real hardware, so wall-clock time must be reported. ProxyBO showed that jacob_cov is positively correlated with accuracy on NAS-Bench-201 and negatively correlated on NAS-Bench-101. That is the sign flip that rules out any single fixed recipe. Can Training Logs Make Model Comparisons More Precise? supplies a warning from a neighboring problem. Its linear, non-pooled adjustment using early training logs made confidence intervals wider at 10 runs, because the adjustment coefficients were themselves estimated from too few runs. Estimating proxy weights from a handful of evaluations should fail in the same way.
The closest method is ProxyBO. It tracks each proxy's Kendall-tau against observed accuracies within a single run, fuses proxies by rank to avoid scale mismatches such as scores of $10^{14}$ against 0.8, and hands control to a Bayesian optimization surrogate on a decaying softmax schedule. We keep its rank transform and its idea of in-run reliability tracking. We replace its heuristic weights with a posterior that carries a cross-task prior and an explicit uncertainty. Transfer NAS with Meta-learned Bayesian Surrogates also transfers across datasets, by meta-learning a deep-kernel Gaussian process over 4,230 ImageNet-derived tasks. It learns point-estimate kernel weights, does not model proxy reliability, and needs a large pre-evaluated meta-corpus. Our population is a handful of benchmark tasks, and we transfer only a few interpretable reliability parameters.
BayesNAS is "hierarchical" in a different sense: its priors on supergraph edges enforce valid connectivity inside one weight-sharing network. It pools nothing across tasks or evaluations. The hierarchical-pooling method itself comes from Estimating Pass@k from Fewer Samples with Hierarchical Bayesian Priors. That work pools success rates across coding tasks with Beta-Binomial priors and reports that 5 samples per task beat the naive estimator at 20. It also shows that boundary-heavy data need specialized priors, which is a caution we carry into the prior design. BANANAS supplies the architecture surrogate we build on: a path-encoded ensemble of multi-layer perceptrons (small feed-forward networks) used for Thompson sampling.
Research Program
For a search task $t$ (a search space paired with a dataset), every candidate architecture $i$ in the pool can be scored cheaply by each proxy $k$. Following ProxyBO, each proxy's scores are converted to normal scores by their rank over the whole candidate pool, giving $s_{tki}$. This removes scale differences without discarding the proxy's ordering. Let $y_{ti}$ be the standardized fully trained accuracy, which is observed only for the few architectures the search has paid to train. The model is
$$s_{tki} = a_{tk} + b_{tk}\, y_{ti} + \epsilon_{tki}, \qquad \epsilon_{tki} \sim \mathcal{N}(0, \sigma_{tk}^2),$$
$$b_{tk} \sim \text{Student-}t_\nu(\mu_{g(k)} + \delta_k,\ \tau^2), \qquad y_{ti} \sim \mathcal{N}(m_t(x_i), v_t(x_i)).$$
Here $b_{tk}$ is proxy $k$'s reliability slope on task $t$, $a_{tk}$ its offset, and $\sigma_{tk}$ its noise. The term $\mu_{g(k)}$ is a mean shared by the proxy's family: gradient-saliency proxies share one family and Jacobian proxies another. The term $\delta_k$ is a proxy-specific deviation, and $\tau$ is the between-task spread. The prior mean $m_t(x_i)$ and variance $v_t(x_i)$ come from the BANANAS-style surrogate over the path encoding $x_i$.
The Student-t prior is deliberate. Exchangeability can fail when a proxy genuinely flips sign across search spaces, as jacob_cov does. Heavy tails let a task's data escape the population mean instead of being over-shrunk toward an unrepresentative average. Parameter count and FLOPs enter as two more "proxies". The model must then show that it is not simply rediscovering "bigger is better".
Because the model is Gaussian given the reliability parameters, the unobserved accuracies integrate out in closed form. Inference over roughly $3K$ task-level parameters is fast Hamiltonian Monte Carlo on CPU. The population hyperparameters are fit once on training tasks and then frozen. On the new task, each full training run updates the posteriors of $a$, $b$ and $\sigma$. It also updates the predictive distribution of every untrained candidate, which combines the surrogate prior with all proxy signals weighted by their current posterior reliability. Thompson sampling from this predictive picks the next architecture to train.
The analysis half of the program asks when pooling helps and when it hurts. We will measure how the posterior over jacob_cov's slope on NAS-Bench-101 moves from a positive population prior to the correct negative value as the number of full evaluations $n$ grows. That trajectory is the most direct test of whether shrinkage adapts, rather than just imposing the average. We will also vary the hyperprior on $\tau$, since the file notes that sensitivity to the dispersion prior is a known weakness when groups are few. With only about six tasks, this sensitivity is a central risk, not a footnote.
Baselines
The naive reference points are random search and Regularized Evolution. NAS-Bench-101 reports that most of its space exceeds 90% CIFAR-10 accuracy, which makes early random search hard to beat, and that Regularized Evolution, BOHB and SMAC reach random search's final performance about 5x faster. A method that does not clearly beat Regularized Evolution in full-training count and in wall-clock time has no claim. All methods will be run on the tabular benchmarks with the same candidate pools, the same 50-evaluation budgets, and at least 20 seeds each.
The strongest existing method is ProxyBO. It reports a 70% test-regret reduction over the strongest baseline on NAS-Bench-201 CIFAR-100, and a 3.92× to 5.41× speedup over Regularized Evolution. We will rerun it with its own temperature settings, the same proxy set, and the same BANANAS-style surrogate, so that the only difference is how proxy trust is set. Beating it means lower median regret at $n \le 20$ full evaluations on held-out tasks, with the gap outside seed variability, and no loss at $n = 50$. I found no reported ProxyBO result on NAS-Bench-NLP or NAS-Bench-301 in the file, so those comparisons rely entirely on our reruns.
BANANAS without proxies isolates what proxies add at all. The zero-cost voting ensemble (Spearman 0.82 on NAS-Bench-201) is the "just use proxies, no search model" reference. A meta-trained predictor serves as a cross-task learned competitor. It follows the OMNI hybrid from How Powerful are Performance Predictors in Neural Architecture Search?, which reports Kendall tau 30% higher than the best single predictor. Here it is implemented as boosted trees over path encodings plus proxy scores, trained on the same held-out-task splits.
The ablation that isolates the key ingredient holds the likelihood and search loop fixed and changes only the prior on $b_{tk}$. The no-pooling version uses flat priors, so each task stands alone. The complete-pooling version sets $\tau = 0$, which gives one reliability per proxy for all tasks. The partial-pooling version is the full model. If partial pooling does not beat both ablations at small $n$, the hierarchical idea has failed, whatever the comparison with ProxyBO shows.
Why not just…?
Why not just track proxy reliability inside each run, as ProxyBO does? With five evaluations, a Kendall-tau estimate has enormous variance, and a softmax over noisy correlations can confidently weight the wrong proxy. The training-log paper's result, where small-sample coefficient estimation widened confidence intervals at $n = 10$, is the analogous failure. The no-pooling ablation and the ProxyBO rerun test this directly. If ProxyBO matches partial pooling at $n = 5$ to $10$, the population prior contributes nothing, and we will say so.
Why not just pick the best proxy or ensemble from past benchmarks and use it everywhere? This is complete pooling, and the jacob_cov sign flip reported by ProxyBO is the counterexample. A fixed recipe learned on NAS-Bench-201 would actively steer a NAS-Bench-101 search toward worse networks. The $\tau = 0$ ablation will show how large that penalty is. If the penalty is small on every held-out task because the proxy set excluding jacob_cov happens to transfer well, the problem is easier than claimed, and a fixed ensemble is the right answer.
Why not just feed proxy scores as features into a boosted-tree predictor trained across tasks? This is the real threat. OMNI-style hybrids already combine proxies with encodings, and boosted trees sit on the Pareto frontier when queries must be instant. A tree model can learn task-dependent proxy use only through task features, however, and with about six training tasks it has little to learn them from. Its predictions also carry no calibrated uncertainty for Thompson sampling. The meta-trained tree baseline settles this. It is run on identical splits and turned into a search policy with a bootstrap ensemble. If it matches us in regret, the contribution reduces to interpretability, and we will report it that way.
Why not just use a meta-learned deep-kernel surrogate like TNAS? TNAS needs thousands of pre-evaluated meta-datasets and learns point estimates. Our setting assumes a population of a handful of benchmark tasks. Where TNAS-style meta-training data exist, it may well win. Our claim is limited to the few-task regime typical of new search spaces.
Proposed Experiments
The main experiment is leave-one-task-out search. The six tasks are NAS-Bench-101 CIFAR-10; NAS-Bench-201 CIFAR-10, CIFAR-100 and ImageNet-16-120; NAS-Bench-301; and NAS-Bench-NLP on Penn TreeBank. The hyperprior is fit on five tasks and the search is run on the sixth. Held-out ranking quality is Kendall tau of the predictive mean against true accuracy at $n \in \{0, 5, 10, 20, 50\}$. The search metric is regret against the known optimum, for example 73.51% on NAS-Bench-201 CIFAR-100 and 94.32% on NAS-Bench-101. Regret is plotted against both full-training count and wall-clock time, with proxy computation and inference time included.
Because the three NAS-Bench-201 tasks share a space and are highly similar, a harder variant holds out an entire search space. This tests whether pooling survives the exchangeability break where it is most likely to fail.
Three checks decide the proposal. First, at $n \le 10$ on held-out spaces, partial pooling must beat both the no-pooling and the complete-pooling ablations. Second, the posterior slope for jacob_cov on held-out NAS-Bench-101 must cross zero within roughly the first ten to twenty evaluations, not stay pinned to the positive population mean. Third, conclusions must be stable across a factor-of-four change in the hyperprior scale on $\tau$. Failing the first check kills the idea. Failing the second means the prior is over-shrinking, and the heavy-tailed or mixture priors need rethinking. Failing the third means six tasks are too few to support a population model, which would itself be a useful negative result for anyone proposing cross-benchmark transfer in architecture search.
References
- Zero-Cost Proxies for Lightweight NAS
- ProxyBO: Accelerating Neural Architecture Search via Bayesian Optimization with Zero-Cost Proxies
- Can Training Logs Make Model Comparisons More Precise?
- Transfer NAS with Meta-learned Bayesian Surrogates
- BayesNAS: A Bayesian Approach for Neural Architecture Search
- Estimating Pass@$k$ from Fewer Samples with Hierarchical Bayesian Priors
- BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search
- NAS-Bench-101: Towards Reproducible Neural Architecture Search
- How Powerful are Performance Predictors in Neural Architecture Search?