Cleaning the Search Set Before Shrinking It: Noise-Gated Data Selection for Cheap Architecture Search

Neural architecture search (NAS) is the automated design of network structures. It has become affordable mainly by searching on small, curated slices of the data. Typical choices are a few thousand "hard" images, a diverse coreset, or samples whose predictions keep flipping during search. The trouble is that every one of these selection rules prefers exactly the examples that mislabeled data looks like: ambiguous, far from the class centre, unstable. On a web-scraped or crowdsourced dataset, the cheap search therefore runs on a subset that is noisier than the dataset it came from. This proposal fixes the order of operations. First, decide which examples are trustworthy, using one dataset-wide cleanliness score of the kind developed in the noisy-label literature. Only then select the small, informative subset the search will use. The score comes from the search model itself, at almost no extra cost.

The core mechanism comes from "global sample selection." This line of work, studied for training a single classifier on noisy labels, judges each example's cleanliness against the whole dataset rather than against the other examples that happen to share its mini-batch. NAS is a setting where this matters more than usual, for two reasons. Search subsets can be as small as 3% of the data, so a few misjudged examples carry heavy weight. And a search network that shares weights across many candidate architectures already produces several "views" of each example for free, through the different sub-architectures that process it. We propose to turn that by-product into a purity gate, apply it separately to the data that trains the weights and the data that scores architectures, and measure whether this restores ranking fidelity at the speed of data-pruned search.

Core Claim

Data-efficient NAS methods choose their subset by an informativeness signal. Accelerating Neural Architecture Search via Proxy Data keeps the low- and high-entropy tails of an auxiliary model's predictions. Core-set Sampling for Efficient Neural Architecture Search uses greedy k-center coverage of pretrained embeddings. Efficient Architecture Search via Bi-level Data Pruning keeps samples with high variance of error over recent epochs. Under label noise, each of these signals is raised by mislabeling: wrong labels give high entropy, sit in outlying regions, and oscillate. Our claim is that the subset must be chosen in two stages. A global, bounded purity gate comes first. Informativeness selection then runs only among examples that passed the gate. With this order, a 5–10% search subset drawn from a noisy dataset should rank architectures about as well as the same budget drawn from clean data, and better than robust-search methods that process the whole noisy dataset.

The new ingredient is where the purity score comes from. In a weight-sharing supernet (one large network whose sub-paths are the candidate architectures), each training step routes an example through some sampled path. We keep, for every example $i$, a running average $s_i$ of the Jensen–Shannon divergence between the routed path's predicted class distribution and the example's label. Jensen–Shannon divergence is a bounded measure of mismatch between two distributions, lying in $[0,1]$, which is why it supports one threshold that stays meaningful across epochs; UNICON and Jo-SRC rely on it for exactly this reason. We also keep a disagreement score $d_i$, the same divergence computed between the two most recent different paths that processed the example. Jo-SRC needs extra augmented forward passes, and co-teaching methods need a second network, to get a consistency signal. Here different architectures in the supernet already disagree on mislabeled or out-of-distribution examples, so the supernet serves as its own committee.

Research Program

The method has three parts.

Warm-up and the gate. The supernet first trains briefly on all data, as DARTS-style searches already do. We then set per-class thresholds on $s_i$ from the dataset-wide distribution. The default keeps a fixed fraction of each class, following UNICON's uniform selection. An alternative uses a mean-plus-$k$-standard-deviations cut from the whole-dataset statistics, as in Group benefits instance for data purification. Examples above the threshold are excluded. Among excluded examples, a low $d_i$ means the paths agree on a different class, which marks in-distribution label errors; we relabel these to the supernet consensus in the weight-training split only. A high $d_i$ marks likely outliers, which are dropped.

Separate gates for the two splits. The gate is applied independently to the split that trains the shared weights and the split whose loss drives the architecture parameters. On Robustness of Neural Architecture Search Under Label Noise found DARTS especially sensitive to noise in the architecture-scoring split. On that split we never relabel, because corrupted architecture gradients are the main thing to avoid.

Shrinking the clean pool. Within the clean pool, we apply bi-level data pruning's variance-of-error criterion, or the entropy-tail rule, to reach the compute budget. The gate and the pruning are refreshed every few epochs, so selection follows the supernet as it changes.

We instantiate this on a single-path one-shot supernet, where path diversity comes for free. We also instantiate it on DARTS, where the mixed supernet has no discrete paths. For DARTS we compute $d_i$ from one extra forward pass on a quarter of each batch through a randomly sampled single operation per edge, and we charge that cost to our method.

We will also test a cheap side claim at the zero-cost end of the cost spectrum. Zero-cost proxies such as those evolved in EZNAS score untrained architectures from gradients on one labeled batch. An Evaluation of Zero-Cost Proxies showed that perturbing the inputs of that batch does not matter, but nobody in our file has corrupted its labels. We will measure how proxy rank correlation degrades with label noise and whether drawing the scoring batch from the gated pool restores it.

Baselines

The naive reference is vanilla DARTS on the full noisy training data. Towards a Robust Differentiable Architecture Search under Label Noise reports that it reaches 69.0% on CIFAR-10 with 50% symmetric noise and collapses toward parameter-free operations (57 parameterless versus 23 convolutions). A second naive point is the unmodified data-efficient searches run on noisy data, at the same subset budget we use: proxy-data entropy tails, k-center coresets, and bi-level data pruning. On clean data, bi-level pruning reports a CIFAR-10 improvement from 97.00% to 97.46% at half the compute, and coreset search reports 61.4% on Tiny-ImageNet with an 8.8x speedup. Beating these on noisy data at equal budget is the minimum bar.

The strongest robust-search competitors are two full-data methods. The first is nConv DARTS from Towards a Robust Differentiable Architecture Search under Label Noise, which reports 85.7% on CIFAR-10 under 50% symmetric noise and 69.8% on Clothing1M. The second is the symmetric Robust Log Loss variant from On Robustness of Neural Architecture Search Under Label Noise, which reports 87.53% in the all-noisy 60% symmetric setting versus 81.62% with cross-entropy. These process the full dataset, so they represent the accuracy ceiling we must match at a fraction of the cost. We will rerun both with their published settings on our noise draws.

Fairness. For every method, the discovered architecture is retrained from scratch with one fixed noisy-label training recipe on the full noisy data. This separates search quality from training-method quality, which matters because the published retraining numbers above mix the two. We report total GPU hours including any selector training, and tune each method's own hyperparameters with the same number of trials.

Beating them means matching nConv's retrained accuracy within noise across seeds while using at most a quarter of its search GPU hours, and exceeding all subset-based methods on ranking correlation at equal budget.

Two ablations isolate the key ingredients. The first reverses the order, pruning first and gating second, to test the ordering claim. The second replaces the supernet's own $s_i, d_i$ with the same gate computed by a small separately trained auxiliary network, to test whether the self-supervising committee is needed.

Why not just…?

Why not just clean the dataset upfront with a cheap filter and then run any efficient search? This is the serious threat. The proxy-data method already trains a small ResNet-20 to compute entropies, so adding a divergence-based filter to that model costs almost nothing. Two arguments favour the supernet as selector. How does a Neural Network's Architecture Impact its Robustness to Noisy Labels? shows that which examples look clean depends on the selecting architecture's inductive bias. Bi-level data pruning argues that static, upfront selection misses how the supernet changes during search. Neither argument is proof, and the auxiliary-selector ablation settles the question directly. If it matches our method, the honest conclusion is that the contribution is the gate-then-prune ordering. That would still be a usable result, and we will report it as such.

Why not just swap in a symmetric loss or nConv and keep the full data? These methods protect accuracy but do nothing for cost, since every epoch still sweeps the whole noisy dataset. The symmetric-loss guarantee in On Robustness of Neural Architecture Search Under Label Noise covers only symmetric noise. Our comparison on real noise and on class-dependent noise tests whether that limit bites. We will also run both combined with our gate. If combining adds accuracy, the methods are complementary rather than competing.

Why not search without labels, as in Are Labels Necessary for Neural Architecture Search? or Neural Architecture Search With Random Labels? Both still train a supernet over the full image collection, so they do not reduce cost. Both also discard the correct majority of labels in realistic noisy data. The random-label angle metric reports a Kendall tau of about 0.5, which we include as a ranking reference on the same benchmark.

Proposed Experiments

The decisive benchmark is NAS-Bench-201 (NAS-Bench-201), which provides clean, fully trained accuracies for all 15,625 cells on CIFAR-10, CIFAR-100 and ImageNet16-120. This lets us measure ranking fidelity exactly while injecting noise only into the search data. We use symmetric noise at 20/50/80%, asymmetric class-flip noise, and, on CIFAR-100, the realistic CIFAR-100N labels.

The metrics are four:

  • Kendall tau between each method's architecture ranking and benchmark clean test accuracy, overall and within the top decile, the regime where EZNAS notes proxies fail.
  • Clean test accuracy of the selected cell.
  • Search GPU hours.
  • Precision and recall of the gate against the known injected corruptions, plus the actual noise rate inside each method's search subset.

The noise rate inside the subset is the most direct check of our motivating claim. We predict that entropy tails, k-center coresets and variance-of-error pruning produce subsets noisier than the base dataset, and that our gate reverses this.

On the DARTS space we then search on CIFAR-10 under 50% symmetric noise and on Clothing1M. We retrain under the fixed recipe and compare against nConv's 85.7% and 69.8%, and the symmetric loss's 87.53%, as rerun under our protocol. We also count parameter-free operations in the discovered cells, to test whether removing noisy examples alone prevents the skip-connection collapse that nConv was built to stop.

The zero-cost side study reports rank correlation of standard proxies on NAS-Bench-201 as the scoring batch's label noise rises, with and without the gated batch.

The idea is wrong, or at best minor, under either of two outcomes. If the auxiliary-selector ablation equals the full method, the supernet committee adds nothing beyond the ordering rule. If ungated subset methods lose no ranking fidelity under 50% noise, the motivating failure does not occur.