Condensing Data from a Supernet to Rank Architectures at High Resolution

Neural architecture search, the automated design of network wiring, is expensive mostly because every candidate network has to be trained before anyone knows how good it is. One obvious shortcut is to train candidates on a tiny synthetic dataset instead of the real one. Dataset distillation produces such a set: it optimizes a few hundred artificial images so that a network trained on them behaves like one trained on millions. The shortcut has two problems. Classic distillation solves a nested ("bi-level") problem that differentiates through an entire inner training run, and that becomes unaffordable at ImageNet resolution. And when the synthetic data is tuned for one network, it ranks other networks badly, sometimes worse than chance.

We propose to fix both problems with one design change. Instead of synthesizing images to match the internal statistics of a single trained teacher, as decoupled methods do at full resolution, we synthesize them to match the statistics of many architectures at once. Those architectures are the sub-networks of one weight-sharing "supernet" (a single over-parameterized network that contains every candidate in the search space as a path). The synthesis step has no inner training loop, so it runs at $224\times224$. Because the target statistics come from the whole search space rather than one model, we expect the resulting data to keep the ordering of candidates intact. Candidates are then trained independently, from scratch, on the condensed set. The supernet supplies the data and never scores the candidates itself.

Core Claim

Rank-faithful condensed proxies do not require the second-order hypergradient machinery that has so far been the only demonstrated fix for rank collapse. A first-order, single-level objective that averages batch-normalization statistic matching over randomly sampled supernet paths should recover most of that rank fidelity, and it should keep working at resolutions where hypergradient methods cannot run. The claim is falsifiable, and we name the result that would refute it below: the gain could disappear once the simpler explanation, teacher soft labels on a small real subset, is controlled for.

Prior Work and Positioning

Three papers motivate the proposal by documenting the gap. Calibrated Dataset Condensation for Faster Hyperparameter Search (HCDC) shows that standard condensation used as a search proxy produces Spearman correlation as low as −0.33 on NAS-Bench-201, because the data overfits its single surrogate network. HCDC repairs this by matching hypergradients, meaning gradients of validation loss with respect to architecture parameters, computed through the implicit function theorem with Neumann-series inverse-Hessian approximations, and reaches 0.74. It is evaluated only on CIFAR and small graphs. Our objective is first-order and contains no Hessian terms. The diversity it needs comes from the set of architectures whose statistics are matched, not from calibrating against architecture gradients.

The cross-architecture studies motivate the same point from the accuracy side. DataDAM matches attention maps of randomly initialized networks, and ATOM shows that data distilled with CNNs transfers poorly to Vision Transformers. Both measure transfer as accuracy on a handful of hand-picked networks, never as rank correlation over a search space.

One paper supplies the method. Squeeze, Recover and Relabel (SRe²L) decouples condensation into three single-level stages: train a teacher, synthesize images by matching the teacher's batch-normalization running means and variances, and relabel the images with the teacher's soft predictions. It reaches 60.8% on ImageNet-1K at 50 images per class at $224\times224$, while being 16–52× faster than trajectory matching and using 6–11× less memory. Its rank behavior across architectures has never been measured. We reuse its recover-and-relabel machinery but replace the single teacher with sampled supernet paths. We also take from Differentiable Neural Network Architecture Search the idea that a whole cell space can live inside one shared network.

Other alternatives to unrolling are adjacent but not used. TESLA keeps trajectory matching at constant memory and is tested at $64\times64$. Distribution Matching matches features under random networks. Both remain single-model or random-model objectives with no architecture-ranking evaluation.

Two papers constrain the design. NAS-Bench-201 reports that architecture rankings inside supernets depend strongly on whether batch statistics or running estimates are used. We therefore recalibrate per-path statistics on real data instead of trusting shared running statistics. Rethinking Performance Estimation in NAS finds that one-shot and low-epoch estimates correlate only locally with final performance, and that training epochs and depth are the hyperparameters that matter most for keeping rankings intact. This finding is the reason the supernet must not do the scoring. It also tells us that our candidates' training schedule on synthetic data must be tuned, not assumed.

Research Program

The pipeline has three stages. First, train a weight-sharing supernet once on the real dataset with uniform path sampling. Second, recover the synthetic images. At each synthesis step, sample a path $a$ uniformly from the search space and recalibrate that path's batch-normalization statistics with a few real batches. Then update the synthetic images $S$ to minimize

$$ \mathcal{L}(S)=\mathbb{E}_{a}\Big[\sum_{l}\big\|\mu_l(S;a)-\hat\mu_l^{a}\big\|_2^2+\big\|\sigma_l^2(S;a)-\hat\sigma_l^{2,a}\big\|_2^2+\lambda\,\mathrm{CE}\big(f_a(S),y\big)\Big]. $$

Here $\mu_l(S;a)$ and $\sigma_l^2(S;a)$ are the batch mean and variance of the synthetic images at layer $l$ of path $a$. $\hat\mu_l^{a}$ and $\hat\sigma_l^{2,a}$ are the recalibrated real-data statistics for that path. $f_a$ is the path's classifier, $y$ is the target class, and $\lambda$ weights the classification term. Each step is one forward and backward pass through a single path, so memory does not grow with any training horizon. Third, relabel the synthetic crops with soft labels averaged over sampled paths.

Candidates are then trained from scratch on $S$ under a short schedule. The schedule is chosen on a small held-out set of architectures, in the spirit of budgeted performance estimation. Candidates are ranked by validation accuracy on real data.

We will also analyze where the method works. We will test whether rank fidelity grows with the number of distinct paths sampled during recovery, which is the mechanism we claim. We will check whether it degrades for operation types that are rare in sampled paths, such as configurations dominated by skip connections. We will measure how the cost of storing soft labels compares with the cost of storing images, since soft labels already erode the compression of SRe²L-style methods.

Baselines

The naive reference point is a random subset of real images at the same images-per-class budget with hard labels, trained under the same short schedule. Any proxy that cannot beat it is not worth synthesizing. We also report the do-nothing reference of random search with no proxy, measured as the regret of the selected architecture.

The strongest condensation baseline on the question of rank fidelity is HCDC. It is the only method shown to fix rank collapse, with Spearman 0.74 on NAS-Bench-201 CIFAR-10 as reported in HCDC. We will run HCDC with its released settings wherever it fits in memory, which we expect to be CIFAR and plausibly ImageNet-16-120. We will report where it stops fitting at higher resolution instead of approximating it. Beating it means matching or exceeding 0.74 Spearman on CIFAR-10 at lower synthesis cost, and producing a valid ranking at $128$ and $224$ resolution where HCDC does not run.

The strongest zero-training baseline is EZNAS-A, with Kendall τ 0.65 on NAS-Bench-201 CIFAR-10 as reported in EZNAS, alongside NASWOT and Synflow at 0.57. Our method must beat these on top-10% precision in particular, since that is their documented weak spot. For SRe²L with a single teacher, no reported result was found on Lacuna for its rank correlation. We run it at identical images-per-class budget, the same synthesis iterations, and teacher capacity matched to the supernet. This baseline is the ablation that isolates the multi-path ingredient.

Two further ablations separate the ingredients. The first is "real subset plus supernet soft labels" at equal storage, which isolates whether synthesis matters beyond relabeling. The second is "multi-path recovery with hard labels", which isolates whether relabeling matters. Finally, we include budgeted low-fidelity training on real data, meaning fewer epochs and lower resolution, with the configuration chosen by the importance-pruning procedure of Rethinking Performance Estimation. It receives the same total GPU hours as our full pipeline, including the cost of training the supernet and synthesizing the data. Every proxy method gets the same tuning budget for its candidate-training schedule.

Why not just…?

Why not just train candidates for a few epochs on downsampled real data? This is the most serious threat. Rethinking Performance Estimation shows that well-chosen low-fidelity settings can cut search cost about 1,000× on CIFAR-10. The same paper finds that cutting epochs damages global ranking and that resolution is among the safer things to cut. That leaves open whether a condensed set beats a tuned low-fidelity schedule at equal compute. We do not argue the point. The equal-GPU-hour comparison decides it, and if low-fidelity training wins at $224$ resolution, we will report that condensation is not the right proxy there.

Why not just use zero-cost proxies? They cost almost nothing. EZNAS, however, reports moderate global correlation and an unresolved inability to separate the top 10% of candidates, which is exactly where search decisions are made. Our evaluation reports top-10% precision and the regret of the selected architecture, so a proxy that only prunes bad models is scored accordingly. Combining the two approaches is also natural: zero-cost proxies prune the candidate pool, and the condensed set ranks the survivors.

Why not just use the supernet's own weights to rank candidates, or reuse an off-the-shelf condensed set? Ranking with inherited supernet weights is the one-shot approach. It shows only local correlation with final performance according to Rethinking Performance Estimation, and its batch-normalization sensitivity is documented in NAS-Bench-201. We include it as a baseline built from the same trained supernet, so any gain is attributable to retraining candidates on condensed data. Off-the-shelf condensed sets collapse to −0.33 Spearman per HCDC, and single-teacher SRe²L is the high-resolution version of that same alternative, which we test directly.

Why not just keep a small real subset and attach teacher soft labels? This plausibly explains much of SRe²L's accuracy. If it ranks as well as our method, the contribution reduces to knowledge distillation on real subsets. The equal-storage ablation exists to detect exactly this, and we will state the outcome either way.

Proposed Experiments

Stage one uses NAS-Bench-201, where all 15,625 architectures have ground-truth accuracy on CIFAR-10, CIFAR-100 and ImageNet-16-120. We report Spearman correlation, Kendall τ, top-10% precision and the regret of the selected architecture, with three seeds and 10 and 50 images per class.

Stage two tests high resolution, where no exhaustive benchmark exists. We use ImageNette and ImageWoof at $128\times128$ and ImageNet-1K at $224\times224$, with the NAS-Bench-201 cell placed in a scaled skeleton. We fully train roughly 50 stratified-sampled architectures per dataset to obtain ground truth. For rank correlation at this scale, no reported result was found on Lacuna for any condensation method. Cost is reported as total GPU hours, including supernet training, so the comparison with budgeted real-data training is honest.

The decisive checks are three. Multi-path recovery must beat single-teacher SRe²L in rank correlation at $224$ resolution. It must beat the real-subset-plus-soft-labels ablation at equal storage. And rank fidelity must rise as more distinct paths are sampled during recovery. If the first check holds and the second fails, the paper becomes a negative result about synthesis for search, which is still worth publishing.