Neural architecture search (NAS), the automated design of network topologies, has become affordable mainly because candidate networks are no longer trained to convergence. They are scored with cheap stand-ins: a few epochs of training, or a single gradient computation at initialization. Every stand-in that looks at data also looks at labels. When those labels come from a crowd that contains lazy spammers and deliberate saboteurs, the cheap score can quietly go wrong. An audit of poisoned NAS found that 50% random label flipping drives a gradient-based search below random architecture selection even though absolute accuracy barely moves (Hard Work Does Not Always Pay Off). The crowdsourcing literature takes a different view of saboteurs. A worker who is systematically wrong carries as much information about the truth as a worker who is systematically right, and the trick is to invert that signal rather than discard it (Learning to love diligent trolls).
This proposal brings that view into cheap architecture search, with one change. Search needs only the ordering of architectures, not correct labels, so the inversion can happen on rankings instead of on labels. We split the crowd into groups of annotators with similar response patterns. We score candidate architectures cheaply and separately on each group's own labels. Then we combine the group rankings with a reliability model that allows negative weights: an adversarial group's ranking can be flipped, a spammer group's ranking can be ignored, and the overall sign is fixed by a proxy that never sees labels. The pitch is that a saboteur who consistently relabels classes does not damage architecture ranking at all unless you mix their labels with everyone else's, and that avoiding the mix is cheaper and more robust than repairing it.
Core Claim
The claim rests on a symmetry. At standard initialization the output units of a classifier are exchangeable, meaning each class's weights are drawn independently from the same distribution. Training on a dataset whose labels have been consistently permuted (class A always reported as B, B as C, and so on, which in binary tasks is exactly the "diligent troll" inversion) therefore produces the same distribution of training trajectories as training on the true labels. Any label-dependent cheap estimator, whether early-epoch held-out accuracy or a gradient saliency score such as SNIP or grad_norm, gives the same expected architecture ranking on a permuted-label group as on an honest one. Harm appears only when permuted and honest labels are pooled on the same images, which produces inconsistent, instance-level noise. Real human noise of this kind is memorized early in training (Learning with Noisy Labels Revisited), and that is exactly the regime where early-stopped estimators operate.
Adversaries who are not permutation-consistent behave differently. Examples are workers who always pick some wrong class, or strategic attackers who target split-decision items. Their group rankings may be uninformative, or they may be systematically anti-correlated with true quality. The negative improvement over random selection reported under poisoning suggests the second case happens. We treat both cases with one model. Let $S_{g,a}$ be the rank-normalized cheap score of architecture $a$ computed only on annotator group $g$'s labels, and $q_a$ the unknown clean-data quality of $a$. We fit
$$S_{g,a} = w_g\, q_a + b_g + \varepsilon_{g,a},$$
where $w_g$ is the group's reliability (positive for honest or permutation-consistent groups, near zero for spammers, negative for anti-informative groups), $b_g$ is a group offset, and $\varepsilon_{g,a}$ is estimator noise. The estimate of $q$ is the leading singular vector of the centered score matrix, optionally computed with a robust variant. This is the same rank-one structure that crowdsourcing work uses to recover worker reliability, surveyed in Learning From Crowdsourced Noisy Labels, but applied to architecture scores instead of item labels.
The model has an unavoidable sign ambiguity: $(w, q)$ and $(-w, -q)$ fit equally well. We fix the sign with a data-free proxy such as Synflow. Such a proxy is weak as a selector but cannot be poisoned through labels. The alternative anchor is a few dozen trusted labels. This choice matters. The "safe-as-majority" anchor used for troll inversion fails completely when its majority premise breaks (Learning to love diligent trolls). A label-blind anchor only needs to be positively correlated with quality, which it is on clean benchmarks at Kendall tau 0.57 (EZNAS).
Research Program
The work has three parts. The first verifies the invariance claim empirically. We sample architectures from NAS-Bench-201, whose clean CIFAR-10 accuracies serve as ground truth. For each, we compute early-epoch accuracy and label-dependent zero-cost proxies on the true labels and on consistently permuted labels. We check that the two rankings agree up to seed noise. We then measure how fast ranking degrades as permuted and honest labels are mixed on the same images.
The second part builds the pipeline. Annotators are clustered using only the worker-by-item label matrix, following latent-class and shared-confusion-matrix ideas; this step is cheap and runs on CPU (Learning to love diligent trolls). Each group gets its own training split and held-out split drawn from items it labeled. The cheap estimator runs per group. The rank-one model above combines the resulting rankings. To match cost, the total GPU time across groups is capped at what a single estimator on majority-vote labels would use. Groups get proportionally fewer epochs or reduced image resolution, which budgeted-estimation work identified as the cheapest knob to turn (Rethinking Performance Estimation in NAS).
The third part stress-tests the method where it should break: groups too small for reliable estimates, adversaries who mimic honest workers on easy items in the manner of SubPac (Black-Box Data Poisoning Attacks on Crowdsourcing), and an adaptive attacker who knows our clustering step.
Crowds are built on CIFAR-10N and CIFAR-100N. The human labels in these datasets supply realistic, instance-dependent honest noise (Learning with Noisy Labels Revisited). On top of them we add simulated worker pools of four kinds: consistent permuters, always-wrong trolls, uniform spammers, and SubPac-style strategic attackers. Adversary fractions run from 10% to 90%, and redundancy runs from one to five labels per item. Each simulated worker labels a random subset of items.
Baselines
The naive reference is majority vote followed by the same cheap estimator and the same search algorithm. This is what a practitioner would do by default. Its expected failure is the mixing effect described above.
The strongest competitor is Dawid-Skene aggregation followed by the same cheap estimator. Dawid-Skene learns a full confusion matrix per worker, so in principle it can already invert consistent adversaries. When workers overlap enough, it may recover nearly clean labels. It runs on identical label matrices and uses an identical estimator budget, and its aggregation time is counted in the cost. Beating it means higher Kendall tau and top-decile precision against the clean NAS-Bench-201 ranking at equal wall-clock cost, in at least the low-redundancy and strategic-attacker regimes. Matching it elsewhere is acceptable.
We also run a trusted-subset baseline. Architectures are trained cheaply on majority-vote labels and ranked on $k$ gold-labelled images, with $k$ from 50 to 1,000. We report the smallest $k$ at which it matches our method, which prices our method in gold labels.
For label-noise-robust search we include the noise-injecting convolution of Towards a Robust Differentiable Architecture Search under Label Noise. That paper reports 85.7% CIFAR-10 accuracy under 50% symmetric noise versus 69.0% for vanilla DARTS. We run it on majority-vote labels at its published search budget. For label-free search we include Synflow alone and a zero-cost voting ensemble. The ensemble reaches Spearman 0.82 on clean NAS-Bench-201 (Zero-Cost Proxies for Lightweight NAS), but part of that ensemble uses labels.
We found no reported result on Lacuna for the rank correlation of any cheap estimator on CIFAR-N or other crowdsourced labels, so every baseline number in this regime is produced here.
Three ablations isolate the key ingredients. The first forces all group weights positive, which removes the inversion. The second uses a single group, which removes the partition. The third removes the data-free sign anchor.
Why not just…?
Why not just aggregate labels with Dawid-Skene and run ordinary cheap NAS? This is the real threat, and we do not argue around it. With dense overlap and consistent adversaries, confusion-matrix aggregation should work, and our method should add little. The case for our method rests on two regimes. In the first, overlap is sparse, and latent-class methods are known to need dense rater matrices (Learning to love diligent trolls). In the second, attackers disguise themselves: SubPac attackers reach reliability scores of 0.75–0.90 under standard aggregators while honest workers sit near 0.80 (Black-Box Data Poisoning Attacks on Crowdsourcing). The redundancy-by-attack grid will show where each method wins. If Dawid-Skene matches us throughout, the paper's contribution narrows to the invariance finding, and we will say so.
Why not just use a label-free proxy like Synflow? It cannot be poisoned through labels, but it is also weak: Kendall tau is 0.57 against 0.65 for evolved data-dependent proxies on clean NAS-Bench-201 (EZNAS). Training-free search gave negligible gains over random selection even without poisoning (Hard Work Does Not Always Pay Off). We therefore use it only to fix the sign of the ranking. The anchor ablation measures how much of the final ranking quality it contributes by itself.
Why not just train each candidate with a noise-robust method? Suppression methods such as noise-injecting convolutions treat corrupted labels as information to be ignored. When most annotators are trolls, little honest signal is left to keep. Their evaluation also used symmetric synthetic noise rather than intentional annotators (Towards a Robust Differentiable Architecture Search under Label Noise). The high-adversary-fraction runs test directly whether suppression holds up where inversion should not degrade.
Proposed Experiments
The first decisive check is the invariance test on NAS-Bench-201 with CIFAR-10. If early-epoch rankings on consistently permuted labels do not match honest-label rankings within seed noise, the core premise fails and the partition idea loses its main justification.
The second check measures Kendall tau and precision within the true top decile against clean NAS-Bench-201 accuracy. We use top-decile precision because that is where zero-cost proxies are known to fail. These are computed for about 1,000 sampled architectures across the full grid of adversary types, adversary fractions, and redundancy levels, on CIFAR-10N and CIFAR-100N, and repeated on ImageNet-16-120 with simulated crowds for transfer. Wall-clock GPU hours are reported rather than FLOPs, because of the documented gap between the two (Zero-Cost Proxies for Lightweight NAS).
The third check is end-to-end. Regularized evolution is driven by each estimator, and the selected architectures are retrained on clean labels. We then compute the improvement over random selection defined in Hard Work Does Not Always Pay Off, whose P-DARTS value of −1.05 under 50% random label flipping on CIFAR-100 is the failure point we aim to reverse. Success means a positive improvement over random at adversary fractions where majority vote and Dawid-Skene go negative, within the same GPU budget.
The final check targets whether the inversion does any work. We test whether fitted group weights are ever significantly negative. If they never are, the troll-inversion component is unnecessary, and the method reduces to partitioned ranking.
References
- Hard Work Does Not Always Pay Off: On the Robustness of NAS to Data Poisoning
- Learning to love diligent trolls: Accounting for rater effects in the dialogue safety task
- Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
- Learning From Crowdsourced Noisy Labels: A signal processing perspective
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Rethinking Performance Estimation in Neural Architecture Search
- Black-Box Data Poisoning Attacks on Crowdsourcing
- Towards a Robust Differentiable Architecture Search under Label Noise
- Zero-Cost Proxies for Lightweight NAS