Warm-Started Short Training: Graph Hypernetworks as Fidelity Amplifiers for Architecture Search

Most of the cost of searching over neural network designs comes from training each candidate design long enough to tell whether it is good. Practical searches cut this cost by training each candidate briefly, for a few epochs instead of hundreds, and ranking candidates by their early accuracy. The cut has a price: early accuracy favors designs that learn fast rather than designs that end up best. This note proposes starting every candidate's short training run from weights predicted by a graph hypernetwork instead of from random weights. A graph hypernetwork is a network that reads a candidate's wiring diagram and outputs a full set of weights for it in one pass. The hope is that a few epochs from a good starting point reveal what many epochs from scratch would. The design is then judged on the right quantity, which is how well the search ranks candidates per GPU-hour, with the cost of training the hypernetwork included.

This differs from the established way of using hypernetworks in search. That approach, introduced by Graph HyperNetworks for Neural Architecture Search, scores candidates directly with the generated weights and no training at all. Zero-training scoring now competes with cheap scoring rules computed at random initialization ("zero-cost proxies") and with cheap learned accuracy predictors, and it is not clear that it beats them. Generated weights have one thing those competitors lack: a starting point for training. The proposal tests whether that starting point improves the low-fidelity evaluation every practical search already runs. It also tests whether the generator brings in a new bias of its own, by favoring designs it happens to know well.

Core Claim

Let $\rho(b)$ be the Spearman rank correlation between candidates' validation accuracy after $b$ epochs of training and their final accuracy after full training (200 epochs, read from the benchmark table). The claim is that warm-starting shifts this fidelity curve to the left. We test whether

$$\rho_{\text{GHN}}(b) \ge \rho_{\text{rand}}(c\,b) \quad \text{for some } c > 1 \text{ over a range of small } b,$$

where $\rho_{\text{GHN}}$ uses hypernetwork initialization and $\rho_{\text{rand}}$ uses standard random initialization. The factor $c$ is the effective speedup in fidelity. We also claim that $c$ stays above 1 after the hypernetwork's training cost is charged to the search. That is plausible only if the generator is small and trained on the target search space and dataset, not a 654M-parameter general model.

The evidence for a head start is suggestive but not about ranking. On ImageNet, Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models? (GHN-3) reports that a ResNet-50 started from predicted weights reaches 43.7% top-1 after one epoch, against 18.2% from random initialization. A head start is not the same as a faithful ranking. If some designs receive a better head start than others, early accuracy measures the generator's familiarity with each design rather than the design's quality. We call this familiarity bias. It is the analogue for warm-starting of the model-size bias that NAS-Bench-Suite-Zero documents for zero-cost proxies. Measuring and correcting familiarity bias is part of the claim, not a side issue.

Research Program

The generator follows the design of Parameter Prediction for Unseen Deep Architectures (GHN-2), which supplies the method. GHN-2 contributes three components. The first is a message-passing network over the candidate's computation graph. The second is operation-dependent rescaling of the predicted weights, so that signals neither explode nor vanish. The third is training on mixed batches of different architectures at once. We train it on the target dataset over architectures sampled from the search space itself, with a compute budget fixed in advance as a fraction of the total search budget. This choice responds to a constraint established by Pretraining a Neural Network before Knowing Its Architecture. That paper found that a generator pretrained on one family of architectures initializes architectures from a different family (ConvNeXt) poorly. It also found that predicted weights lack diversity across layers, which hurts later training, and that adding noise to the predicted weights partly repairs this. We therefore treat noise injection as a tunable component of the warm-start, with its scale chosen on a calibration set.

Two pieces of analysis sit on top of the generator. The first measures fidelity curves $\rho(b)$ for $b \in \{0, 1, 2, 4, 8, 12\}$ epochs under each initialization. The point $b=0$ recovers the original zero-training use of hypernetworks for scoring. The second is a familiarity correction. We train both initializations on a small calibration subset of candidates. On that subset we fit the warm-start's accuracy advantage at budget $b$ as a function of the generator's own held-out training loss on each architecture. During search, each candidate's score is its warm-started accuracy minus this predicted head start:

$$s_b(\alpha) = \text{acc}_b^{\text{GHN}}(\alpha) - \hat g_b(\ell_\alpha),$$

where $\alpha$ is the candidate, $\ell_\alpha$ is the generator's loss on that candidate, and $\hat g_b$ is the fitted head-start predictor. The search procedures that consume these scores are regularized evolution and successive halving. Regularized evolution was the most robust black-box optimizer in NATS-Bench and NAS-Bench-Suite. Successive halving trains many candidates briefly and spends more epochs on the survivors.

The setting is the topology space of NAS-Bench-201, which has 15,625 cells. Final accuracies are given there for CIFAR-10, CIFAR-100, and ImageNet-16-120. The secondary setting is the size space of NATS-Bench, which has 32,768 channel-width configurations and tests whether a single generator handles changing tensor shapes. The tables only store results of training from random initialization, so all short warm-started runs must be trained by us. The tables supply the ground-truth final accuracy. A stratified sample of about 1,000 cells per dataset, each trained to 12 epochs under every initialization condition, is affordable for cells this small.

Baselines

The naive reference point is short training from random initialization, used inside the same regularized evolution and successive-halving searches with an identical epoch schedule. If the warm-start cannot beat this curve after its own training cost is charged, the proposal fails.

The strongest competitor for ranking alone is a tuned gradient-boosted tree predictor, XGBoost, given zero-cost proxy scores as features. NAS-Bench-Suite reports that tuned XGBoost was the most robust performance predictor across 25 benchmark settings, with an average rank of 1.23. NAS-Bench-Suite-Zero reports that adding zero-cost proxies to such predictors raises Spearman correlation by up to 42%, and that nwot and flops were the most robust single proxies. The file does not give a Spearman value for nwot on NAS-Bench-201 CIFAR-10 (no reported result found on Lacuna for it), so we recompute it from that suite's precomputed scores. The predictor will be trained on the same number of fully labeled architectures that the calibration set uses, with an equal hyperparameter-tuning budget. Under a matched total GPU budget, beating it means a higher top-k overlap and a higher final accuracy of the selected architecture. The final accuracy is read from the table and compared against the space's best values of 94.37% on CIFAR-10, 73.51% on CIFAR-100, and 47.31% on ImageNet-16-120.

The baseline that isolates our key ingredient is supernet weight inheritance. A weight-sharing supernet in the style of Efficient Neural Architecture Search via Parameter Sharing is trained with exactly the GPU-hours given to the hypernetwork. Each candidate is then initialized from its slice of the supernet before the same short training. A supernet is a fixed lookup table of weights. A hypernetwork computes weights conditioned on the whole graph. If the two give the same fidelity curve, graph conditioning adds nothing. This is the decisive ablation.

Two further references come from earlier hypernetwork work. The first is zero-training hypernetwork scoring as in Graph HyperNetworks for Neural Architecture Search. On CIFAR-10 it reported a Pearson correlation of up to 0.68 with converged accuracy and a search cost of 0.84 GPU days. This is the $b=0$ point of our curves. The second is an off-the-shelf GHN-2 pretrained on DEEPNETS-1M, used without any training on the search space. It tells us whether training the generator on the target space is necessary.

Why not just…?

Why not just rank with zero-cost proxies or an XGBoost predictor and skip training altogether? If the only output wanted is a ranking, this is a real threat, and the experiments may show that it wins. Our response is that a search must also train the architecture it chooses, and short evaluations of the leading candidates are where practical searches spend their budget. The comparison is therefore made on total GPU-hours to reach both a selected architecture and a trained checkpoint. We also test a hybrid in which proxy-based predictors prune the pool and warm-started short training ranks the survivors. If the predictor alone matches the hybrid at equal cost, we will report that result.

Why not just inherit weights from a trained supernet? This alternative tests most directly whether a generator is needed. NATS-Bench reports that weight-sharing methods struggle to rank the best architectures, and NAS-Bench-201 documents their collapse toward skip connections. Both findings, however, concern ranking with shared weights and no further training, not initialization followed by brief training. Brief training could wash out the damage from shared weights. The matched-compute inheritance baseline described above settles this question directly.

Why not just train each candidate a few more epochs from random initialization? This is the naive curve. The proposal is justified only if $c$ exceeds 1 after the cost of training the generator is charged. We report the break-even number of candidates beyond which the generator pays for itself.

Proposed Experiments

The first decisive check compares fidelity curves. We plot $\rho(b)$ and the overlap of the top 1% under random, inherited, pretrained-GHN, and space-trained-GHN initializations, with and without noise injection, on all three NAS-Bench-201 datasets. From these curves we read off the effective speedup $c$.

The second check measures familiarity bias. We train the generator on a deliberately skewed subset of cells, for example cells with few convolutions. We then test whether warm-started rankings over-promote held-out cells that resemble that subset, and whether the corrected score $s_b$ removes the distortion. Rank error will be correlated with the generator's loss before and after correction.

The third check is an end-to-end search. Regularized evolution and successive halving each run under fixed total budgets, with ten seeds per budget, and we record table accuracy of the selected architecture against GPU-hours, all costs included.

The fourth check applies the protocol to the NATS-Bench size space. This tests whether one generator produces useful warm-starts across channel widths from 8 to 64. Width-agnostic generators such as NNiT have so far been shown only on four-layer MLPs.

The proposal succeeds if $c$ remains clearly above 1 after costs are charged, if the hypernetwork beats supernet inheritance at matched compute, and if the familiarity correction keeps the warm-start from rewarding the generator's training distribution.