Cheaper Architecture Search Hands More Control to Whoever Games the Data Filter

Neural architecture search (NAS), the automated design of network topologies, became affordable mainly by looking at less data. Modern search methods score every training example on the fly and throw most of them away, or they pick a tiny subset of "informative" examples before search starts. Each of these filters is a per-example data valuation: it decides which examples matter enough to shape the architecture. The data-valuation literature has shown that such scores can be gamed. Small, hard-to-see input perturbations inflate an example's apparent importance by several hundred percent. This note argues that the two facts compound. The harder a search method prunes to save compute, the fewer examples decide the architecture, and the larger the share of that deciding set a strategic data contributor can capture.

We propose to measure this trade-off between search cost and manipulability directly, in a setting where several parties pool data for a shared search and one of them perturbs its data. The attacker has two goals. It wants its examples kept, which inflates its credit whenever retention or influence is used to pay contributors. It also wants to steer the discovered architecture. We then test a simple defense. It takes the retention budget out of the per-example score, which the attacker can manipulate, and computes it at the contributor level, where manipulation is much harder. Within each contributor, it randomizes which examples survive.

Core Claim

In data-pruned NAS, an adversarial contributor's share of the retained search data, and its influence over the chosen architecture, grows as the compute budget shrinks. Full-data search dilutes the attacker. Aggressive dynamic pruning concentrates it. Single-minibatch scoring at initialization is the extreme case. The claim is falsifiable. If the attacker's retention share and the damage to search quality stay flat as the pruning ratio increases, the claim is wrong.

The second claim is that the fix does not require expensive truthful valuation. Setting per-contributor quotas from a contributor-level score, then drawing examples at random within each quota, should recover clean-search quality while keeping most of the speedup. Contributor-level scoring is the idea behind Data Overvaluation Attack and Truthful Data Valuation, which values whole client datasets and is thereby immune to subset-level manipulation.

Where This Sits

Three lines of prior work motivate the problem. The first is the pruning methods we attack. Accelerating Neural Architecture Search via Proxy Data selects a fixed subset from the high- and low-entropy tails of a pretrained ResNet-20's predictions. It cut DARTS (differentiable architecture search) on CIFAR-10 to 40 minutes using about 3% of the data. Efficient Architecture Search via Bi-Level Data Pruning (BDP) prunes training and validation data during search using each sample's variance of error over a short window. It reports over 50% search-cost reduction on NAS-Bench-201 and a CIFAR-10 improvement from 97.00% to 97.46%. Both methods assume honest data, and neither has been evaluated against a contributor who shapes those scores.

The second line supplies the attacks. Adversarial Attacks on Data Attribution introduces two attacks and reports that they transfer across influence functions, TRAK, and Data Shapley. The Shadow attack imitates valuable gradient traits using shadow models and raises compensation by 200–456%. The Outlier attack pushes points into sparse feature regions and raises compensation by 185–643%. We adapt both to entropy and variance-of-error scores. Some attacks do not transfer to our setting. The model-side manipulation of Influence-based Attributions can be Manipulated assumes the adversary controls the model. In our setting the server runs the supernet, so we restrict the attacker to data.

The closest existing work is Hard Work Does Not Always Pay Off: On the Robustness of NAS to Data Poisoning. It shows that poisoning can make an expensive search no better than random, and introduces ΔImp, the percentage-point gain of the searched architecture over a random one from the same space. We reuse ΔImp. That paper studies destructive, mostly label-flipping poisoning on full-data search, and it does not vary the compute budget. We study valuation gaming of the cost-saving filters themselves, with clean-label feature perturbations, as a function of how much data the search keeps. Its finding that training-free TE-NAS is immune to label flipping says nothing about feature perturbations. That question is our third regime.

Two further papers mark the boundaries of the question. Aequa rewards contributors with wider sub-networks and assumes contribution scores are honest; our threat model is the reason that assumption matters. OpenDataVal notes that valuation benchmarks lack adversarial settings, and we contribute one.

Research Program

The setting pools CIFAR-10, CIFAR-100, and ImageNet-16-120 across ten contributors. We use both IID splits and splits skewed by class. One contributor, holding 10% of the data, is adversarial under an L∞ perturbation budget and is not allowed to change labels. All search runs on NAS-Bench-201 (Dong and Yang). Its exhaustive tables give exact final accuracies for any derived cell, so thousands of attacked searches can be scored without retraining.

We study three search regimes of decreasing cost. The first is DARTS with BDP at retention ratios from 100% down to 10%. The second is proxy-data DARTS at subset sizes from 20% down to 3%. The third is zero-cost proxy search, which ranks architectures from a single minibatch (Zero-Cost Proxies for Lightweight NAS); here the attacker's perturbed examples may land in the scoring batch.

We implement two attacker goals.

  • Retention inflation. The attacker maximizes its fraction of surviving examples.
  • Architecture steering. The attacker pushes search toward a target cell, for example one dominated by skip-connections or one tuned to its own classes.

We port two attacks, following the Wang et al. recipes.

  • Black-box Outlier attack. It aims for high-entropy, sparse-region points.
  • Gray-box Shadow attack. A shadow supernet trained on the attacker's own data guides perturbations toward whichever variance-of-error profile BDP's rule retains.

The defense, which we call contributor-capped randomized retention, has two parts.

  • Quotas. Each contributor's share of the retained set is proportional to a contributor-level score, computed once per epoch on a small trusted validation set in the style of FedIF's update–gradient alignment.
  • Random draws within quotas. Within each quota, examples are sampled at random with probability weighted by BDP's score. The weighting is clipped so that no single example is guaranteed to survive.

The product of this program is a set of curves showing attacker share and ΔImp against search GPU-hours, with and without the defense.

Baselines

Unpruned search. The do-nothing reference is full-data DARTS with the adversary present. It costs the most and dilutes the attacker most. It tells us how much of any damage is caused by pruning rather than by the perturbations themselves. We run it with the same perturbed pool, seeds, and search schedule as every pruned variant.

BDP. The strongest existing efficient method is BDP. It is the most recent dynamic pruner, it prunes both the weight-training data and the architecture-validation data, and it reports over 50% search-cost reduction on NAS-Bench-201 and 97.00%→97.46% on CIFAR-10 with DARTS. We found no reported result on Lacuna for BDP under adversarial contributors. We therefore first reproduce clean BDP on NAS-Bench-201 and confirm that its speedup holds, then attack it at matched retention ratios. The defense beats BDP if it keeps ΔImp within the clean-BDP seed spread under attack, keeps at least 80% of BDP's wall-clock saving, and holds the attacker's retained share within one standard deviation of its data share.

Proxy-data selection. We run it at the published 3% operating point and at larger subsets, under the same perturbation budget.

Zero-cost ensemble. The Synflow, jacob_cov, and SNIP voting ensemble reaches Spearman 0.82 on NAS-Bench-201 in Abdelfattah et al. We measure how far that correlation falls when the scoring batch contains attacker points.

Sanitization defenses. We include loss-based sanitization and diffusion denoising, which the NAS-poisoning audit reports fail to restore ΔImp.

Ablations. Two ablations isolate the key ingredient: quotas without within-contributor randomization, and randomization without quotas. Every method gets the same GPU-hour cap and the same hyperparameter-tuning budget of 20 trials on a clean held-out split. Nothing is tuned on attacked runs.

Why not just…?

Why not just search on trusted outside data? P-DARTS searched on clean MNIST reached 97.14% on CIFAR-10, versus 97.32% when searched on CIFAR-10 itself. This is a real threat to the steering half of the proposal. If architectures barely depend on the search data, steering is weak. That result is reported as absolute accuracy, whereas the same audit shows ΔImp can swing by up to 156% even when raw accuracy moves by less than 1%. We therefore run OOD-searched NAS-Bench-201 cells as a baseline measured in ΔImp, on CIFAR-100 and ImageNet-16-120 as well as CIFAR-10. If OOD search matches in-distribution ΔImp everywhere, we will report steering as a non-issue and narrow the contribution to credit inflation. Outside data cannot value contributors' actual data, so the credit-inflation question survives either outcome.

Why not just stratify pruning by contributor, as BDP already stratifies by class? A fixed per-contributor quota may well eliminate retention inflation on its own, which is why it is our first ablation. We expect it to fail at steering. Within a fixed quota the attacker still chooses which of its points survive and what gradients they carry, and a fixed quota also rewards contributors who upload low-value data. If quotas alone restore ΔImp, that is the finding. We would then recommend quotas and drop the randomization component.

Why not just use Truth-Shapley? It is provably robust to subset manipulation, but it requires utility evaluations over coalitions of contributors. OpenDataVal shows Shapley-style methods need 1,000+ training runs, and the authors list reducing its cost as open. Stacking that on search defeats the purpose of cheap NAS. We borrow its contributor-level principle instead, at the cost of one alignment score per contributor per epoch.

Why not just use label-free training-free NAS? The poisoning audit found that TE-NAS resists label flipping but barely beats random selection even on clean data. Its robustness to feature perturbation is untested, and our zero-cost regime tests exactly that.

Proposed Experiments and Decisive Checks

The first check tests the core claim directly. We plot the attacker's retained-share ratio and ΔImp against search GPU-hours across the three regimes. If the ratio does not rise as retention falls, for example from about 1× at full data toward several times at 3–10% retention, the premise fails.

The second check measures credit harm. Honest contributors' retention-based credit, and their FedIF-style scores, should shift under attack. We report the transfer of credit to the attacker in percent, alongside the 77–643% inflation range reported for fixed-architecture valuation.

The third check is the defense frontier. We plot ΔImp and attacker share against GPU-hours for the full defense, each ablation, the sanitization baselines, and OOD search. The defense counts as a success only if it dominates BDP under attack at matched cost.

The fourth check is adaptive-attacker robustness. An attacker that knows the quota rule concentrates its perturbation budget on the few points most likely to be drawn. We report how much each defensive component loses when the attacker adapts in this way.

All results use five seeds and the NAS-Bench-201 lookup tables. We report ΔImp, test accuracy of the derived cell against the space optima (94.37%, 73.51%, and 47.31% on CIFAR-10, CIFAR-100, and ImageNet-16-120), and wall-clock search time.