Which Synthetic Images Make Architecture Search Cheap, and Which Make It Lie

When neural architecture search (NAS), the automated design of network structures, ranks thousands of candidate networks, the expensive part is training each candidate long enough to tell good from bad. A popular shortcut is to train candidates briefly on a tiny synthetic dataset produced by dataset distillation, an optimization that compresses a large training set into a few dozen artificial images. Those images are known to look like texture and noise to humans, and they are known to "overfit" to the small network used to create them. Our proposal is to treat a distilled set as a population of individual images and to measure, image by image, whether each one helps or hurts the proxy's ability to order architectures correctly. We expect a small, recognizable subset of images to carry most of the ranking signal. We also expect the images that distort rankings to share visible traits: high-frequency content, pixel values far outside the natural range, and a preference for candidate cells that resemble the network used for distillation. If that holds, interpretability becomes a selection rule. Removing the harmful images should make the proxy both cheaper, because there are fewer samples to train on, and more faithful.

The opportunity matters because the two communities currently hold complementary halves of an answer. Interpretability work has found that distilled images behave like "semantic atoms," each encoding a stable sub-concept, and that their value is concentrated in early training. NAS work has a concrete, cheaply checkable notion of usefulness, rank correlation with true accuracy, but treats the proxy dataset as a black box. Joining them lets us ask a question neither has posed: which parts of a synthetic dataset carry information about architectures, and can we recognize those parts without expensive calibration?

Core Claim

The ranking fidelity of a distilled NAS proxy is unevenly spread across its samples. Per-sample contributions to rank correlation can be estimated cheaply on a small calibration set of architectures, and they are predicted by interpretable image properties. A pruned proxy that keeps only rank-faithful samples will rank held-out architectures in NAS-Bench-201 better than the full distilled set and better than equally sized random or heuristic subsets, at lower wall-clock cost. The second and riskier claim is that a calibration-free property of each image, namely how consistently its gradients agree across two surrogate networks of different families, can replace the calibration step. That would make the method usable on search spaces where no ground-truth accuracies exist.

Relation to Prior Work

Three papers already use synthetic data to speed up NAS, and our proposal is defined by how it differs from each. Generative Teaching Networks meta-learns a generator whose "alien" images train a learner in 32 steps and uses them as a fast evaluator. Its authors concede that the data gives no insight into what is learned, and they offer no account of why it transfers across candidate architectures. Calibrated Dataset Condensation (HCDC) shows that ordinary condensed sets can produce rank correlations as low as -0.33. It fixes this by optimizing a synthetic validation set to match hypergradients, the gradients of validation loss with respect to architecture choices, and reaches Spearman 0.74 on NAS-Bench-201 CIFAR-10. It never examines what the resulting images contain. DaSS feeds fixed random noise through networks to build embeddings for an accuracy predictor, so its probe inputs are not task data at all. All three optimize or use the synthetic set as a whole. We instead decompose an existing set, attribute ranking fidelity to individual samples, and explain the attribution. Our method is a post-hoc analysis and pruning step that can be applied to HCDC's or a gradient-matching method's output, so it complements rather than replaces them.

Two lines of work supply the method and the motivation. What is Dataset Distillation Learning? supplies the tools: influence-function analysis, multimodal-model captioning, and the findings that distilled pixels leave the [0,1] range, that clipping hurts, and that each image maps to a reproducible sub-concept. We reuse these tools but point them at a new target, architecture ranking instead of single-model accuracy. ATOM and Data-to-Model Distillation motivate the hypothesis by documenting that distilled data collapses when moved to a different architecture family. Those papers treat this as a property of the whole set. We test whether it is concentrated in specific samples.

Two papers constrain the design. The zero-cost proxy results in Zero-Cost Proxies for Lightweight NAS set a high bar: a vote of synflow, jacob_cov, and snip reaches Spearman 0.82 on NAS-Bench-201 with no training at all. NAS-Bench-Suite-Zero shows that feeding such proxies into a learned surrogate model improves correlation by up to 42%, and it warns that many proxies mostly track parameter count. A synthetic-data proxy that cannot match or add to these is not worth its distillation cost, and our evaluation is built around that test.

Research Program

We start from distilled CIFAR-10 and CIFAR-100 sets at 10 images per class, produced by gradient matching, distribution matching, and trajectory matching with the standard small ConvNet surrogate. We also prune HCDC's synthetic validation set. Each candidate architecture from NAS-Bench-201 is trained for a short fixed budget on the proxy, and its proxy score is final proxy validation accuracy. The proxy's quality is the Spearman correlation between proxy scores and the tabulated true accuracies over a set of architectures.

To attribute ranking fidelity to samples, we use a datamodel-style estimate, which fits a linear model predicting an outcome from which training samples were included. For a calibration set of $K$ architectures, we draw $M$ random half-subsets of the distilled set, train every calibration architecture briefly on each subset, and record the resulting Spearman correlation $\rho_m$. We then fit

$$\rho_m \approx \beta_0 + \sum_{i} \beta_i z_{m,i},$$

where $z_{m,i}\in\{0,1\}$ indicates whether synthetic image $i$ was in subset $m$, and $\beta_i$ is image $i$'s estimated contribution to rank fidelity. With 100 images, $K=200$, and $M=200$, this takes 40,000 trainings of roughly 50 steps each on 50 images, which is a few GPU-hours on NAS-Bench-201's small networks. All of this cost will be charged to the method. We will also fit a second, finer attribution: each image's effect on the proxy score of individual architectures, regressed on the architecture's operation counts. This tests whether harmful images specifically inflate cells rich in 3x3 convolutions, the surrogate's own inductive bias.

The interpretive stage asks what distinguishes images with high $\beta_i$ from those with low $\beta_i$. We will measure each image's spectral energy above a frequency cutoff, its fraction of out-of-range pixels, whether a real-data-trained classifier labels it correctly, its multimodal-model caption, and its sub-concept cluster under influence functions, following What is Dataset Distillation Learning?. We will also compute a calibration-free score, the cross-surrogate agreement. This is the cosine similarity between the gradient that image $i$ induces in a randomly initialized ConvNet and the gradient it induces in a structurally different network, a small MLP-Mixer or ViT, averaged over random initializations. The hypothesis is that images whose learning signal survives a change of architecture family are the ones that rank architectures fairly.

The pruning stage keeps the top fraction of images by $\beta_i$ (calibrated variant) or by cross-surrogate agreement (calibration-free variant). We sweep the kept fraction and report correlation and wall-clock time.

Baselines

The naive reference point is the full, unpruned distilled set used as a short-training proxy under the identical step budget. Its correlation tells us whether pruning helps at all. HCDC reports that ordinary condensed sets can fall as low as -0.33 Spearman on NAS-Bench-201 CIFAR-10, so we expect this baseline to be weak. We still rerun it under our protocol rather than quote the figure.

The size-matched controls isolate the key ingredient, which is the per-sample selection rule. A random subset of the distilled set and a random set of real CIFAR images of the same size will be trained with the same steps and optimizer. A heuristic filter that drops the most high-frequency and most out-of-range images, using no attribution, will also be included. If the heuristic filter matches attribution-based pruning, the interpretive finding is real, but the attribution machinery is unnecessary, and we will say so.

The strongest training-free competitor is the zero-cost vote of synflow, jacob_cov, and snip from Zero-Cost Proxies for Lightweight NAS, which reports Spearman 0.82 on NAS-Bench-201. We will recompute it on exactly our held-out architecture sets so that the comparison uses the same architectures. Beating it outright counts as a clear win. Matching it while being complementary also counts, in a narrower sense; complementarity is defined below.

The strongest synthetic-data competitor is HCDC (Calibrated Dataset Condensation), which reports Spearman 0.74 on NAS-Bench-201 CIFAR-10 and cuts search time from 229 to 35 seconds. We will run its released procedure on the same splits, then apply our pruning to its output. Beating HCDC means a higher held-out Spearman at equal or lower total time, with our calibration cost included.

Because our calibrated variant consumes $K$ labeled architectures, the fair labeled competitor is a learned surrogate in the style of NAS-Bench-Suite-Zero: gradient-boosted trees trained on the same $K$ architectures, using all thirteen zero-cost proxies as features, with an equal hyperparameter tuning budget. That paper reports a relative gain of up to 42%, but no absolute NAS-Bench-201 Spearman for this configuration appears on Lacuna, so we will produce the number ourselves. We will also add our pruned-proxy score as one more feature. A gain over the thirteen-feature surrogate is our definition of complementarity. For Generative Teaching Networks, no reported NAS-Bench-201 rank-correlation result was found on Lacuna, so it is included only if a reimplementation is feasible within budget. Short training on real data in the style of EcoNAS, reported at 0.61 Spearman in Zero-Cost Proxies for Lightweight NAS, serves as the low-fidelity reference.

Why not just…?

Why not just use the zero-cost proxy ensemble, which already reports 0.82? This is the most serious threat. The reported HCDC number of 0.74 is below it, so an unpruned synthetic proxy probably loses. The case for synthetic data rests on the known weaknesses of zero-cost proxies: their bias toward large models and their degradation on structurally diverse architectures, both documented in NAS-Bench-Suite-Zero and Zero-Cost Proxies for Lightweight NAS. We will test both directly. We will report correlation within parameter-count bins, where size bias cannot help, and on the cross-search-space transfer below. If the pruned proxy neither beats the ensemble in those settings nor adds to the surrogate, the contribution becomes an explanation of why synthetic proxies fail rather than a better proxy. We will report it that way.

Why not just use HCDC, which was built to preserve rankings? HCDC changes how the whole set is optimized, but it keeps a single proxy model during condensation. Its authors list heterogeneous search spaces as open. If architecture lock-in lives in specific samples, HCDC's output will contain such samples too, and pruning should improve it. The experiment that settles this is applying our attribution to HCDC's set. If every HCDC image has similar $\beta_i$ and pruning gives nothing, HCDC has already solved the problem we target, and our contribution is limited to vanilla distillation methods.

Why not just select a small coreset of real images, which needs no distillation and is readable by eye? The size-matched real-image control answers this directly. Prior evidence suggests real subsets carry less early-training signal per sample (What is Dataset Distillation Learning?), and short training on real data reaches only 0.61. If a real coreset of the same size matches the pruned distilled proxy, the honest conclusion is that distillation buys nothing for ranking, and we will report that.

Why not just filter images by obvious traits like high frequency, without fitting attributions? This is the heuristic-filter ablation. A positive result for it would be welcome, because it would make the method nearly free. In that case, the attribution study's role shrinks to justifying the heuristic.

Proposed Experiments

All ranking is evaluated on NAS-Bench-201 (NAS-Bench-201). The calibration architectures are disjoint from 2,000 held-out architectures, and results are repeated across five calibration draws and three distillation seeds. Metrics are Spearman and Kendall correlation on held-out architectures, the correlation within parameter-count bins, and top-10 regret, meaning the gap between the true accuracy of the best architecture the proxy picks and the table optimum of 94.37% on CIFAR-10, 73.51% on CIFAR-100, and 47.31% on ImageNet-16-120. We also report total wall-clock time including distillation and calibration, measured on the same GPU. Measured time matters because FLOP savings do not translate into wall-clock savings, as Zero-Cost Proxies for Lightweight NAS documents.

Three checks decide the proposal. First, concentration: if the attribution weights $\beta_i$ are not heavy-tailed, so that dropping the lowest 30% of images changes held-out correlation by less than seed noise, the core premise fails. Second, mechanism: the operation-count regression should show low-$\beta$ images inflating scores of 3x3-convolution-heavy cells. Their high-frequency energy and out-of-range fraction should separate them from high-$\beta$ images, with an area under the ROC curve well above chance. If low-$\beta$ images are instead indistinguishable, the interpretability story is wrong even if pruning helps. Third, portability: the calibration-free cross-surrogate agreement score should recover most of the calibrated variant's gain. We test this by pruning CIFAR-10 distilled sets once and then ranking NAS-Bench-101 architectures, a different search space on the same images, without recalibration, using ground truth from the NAS-Bench-Suite-Zero tabulations. Success on this transfer, compared against the zero-cost ensemble recomputed on the same NAS-Bench-101 architectures, is the result that would make the method useful beyond a single benchmark.