Differentiable architecture search trains one large "supernet" in which every connection holds a weighted blend of candidate operations (convolutions, pooling, identity shortcuts). After training, it keeps the operation with the largest weight on each connection. The method has a known failure: it drifts toward parameter-free operations, mostly identity shortcuts, because they help a half-trained supernet immediately and convolutions do not. Hardware-aware versions add a speed penalty to the search objective. That penalty is usually computed from arithmetic counts (FLOPs), and FLOPs say shortcuts and pooling are nearly free and that zeroing 80% of a convolution's weights makes it 5× cheaper. Neither is true on real chips. Shortcuts and pooling are memory-bound, so their cost comes from moving data rather than doing arithmetic. Unstructured sparse weights, meaning zeros scattered at random positions, usually run no faster than dense weights. We argue that a FLOPs-based penalty does more than give a bad speed estimate. It adds a second bias toward the same operations that already enjoy an unfair training advantage, so the two biases compound.
We propose a search in which every candidate is charged a measured, device-specific latency. Each convolution choice is also extended with an execution mode: dense, 2:4 structured sparse (the hardware-supported pattern with exactly two nonzeros in every four weights), or unstructured sparse. The three modes share one set of weights, so a sparse variant never enters the competition as a separately untrained operation. The study has two parts. The first is a diagnosis: how much of the collapse seen in latency-penalized search is caused by mis-measured cost? The second is a method: per-layer, per-device choices of operation and sparsity mode that sit on a better measured-latency/accuracy frontier than searching first and pruning afterward.
Core Claim
Three roles define how this note uses the directions. Operation selection bias is the failure we study. The FLOPs-latency gap is the mechanism we believe amplifies it. Unstructured sparsity overhead is the clearest case where a FLOPs-style cost model rewards the wrong choice. Our claim has two parts, each testable. First, at matched mean latency, a search penalized by FLOPs ends with more parameter-free operations and lower accuracy than the same search penalized by measured latency, and the gap is largest on devices where shortcuts and pooling are expensive relative to convolutions. Second, once sparse execution modes are candidates and are charged measured cost, the search chooses 2:4 sparsity where the hardware accelerates it and dense execution where it does not. It almost never chooses unstructured sparsity on GPUs. The resulting models dominate the "search dense, then sparsify" pipeline wherever the best sparsity pattern differs across layers or devices.
Relation to Prior Work
Work that explains the bias. Fair DARTS shows that softmax competition locks in an early advantage for weight-free operations. DARTS- argues that shortcuts win because they stabilize supernet training, and moves that role into an auxiliary branch. R-DARTS ties collapse to sharp minima of the validation loss. All three study the accuracy-only objective. None asks what an added efficiency penalty does to the bias, and that is the question we add.
Work that adds latency to differentiable search. LA-DARTS adds a learned latency predictor to DARTS but keeps the standard relaxation, which collapses. TF-NAS handles collapse under latency constraints by sampling two paths and removing shortcuts from the candidate pool. DEff-ARTS models cycle counts for a single digital signal processor. MODNAS learns Pareto fronts across 19 devices from pretrained predictors. All of these assume dense execution. None separates how much collapse comes from the cost model, and none treats sparsity as a per-layer, per-device execution choice.
Work that supplies the sparsity methods and limits. Zhou et al. train 2:4 networks from scratch with SR-STE, a gradient trick that stops pruned weights from oscillating across the threshold. They use fixed architectures. Hu et al. show that dense non-matrix operations cap end-to-end 2:4 speedup at 1.21×. That is an Amdahl's-law limit we expect our measured costs to reproduce. Kübler et al. give a proximal operator for post-training 2:4 pruning. Mostafa & Wang concede that their unstructured sparse networks gain no wall-clock speed.
Work that motivates the measurement approach. ShuffleNet V2 shows that element-wise and memory-heavy operations cost far more time than their FLOPs suggest. HW-NAS-Bench reports that the rank correlation between FLOPs and latency falls as low as 0.18 on an Edge TPU. Its measured latencies are what make the diagnostic part of this study cheap.
Research Program
Part one: diagnosis, run on NAS-Bench-201. The tabular ground truth is exact here. From the measured latencies of all 15,625 architectures in HW-NAS-Bench, we fit an additive cost model for each device, with one cost per operation per edge. Because the fit includes intercepts for shortcuts and pooling, those operations receive nonzero, device-specific prices. We report the fit's rank correlation with true latency, so that any failure of additivity is visible rather than hidden. We then run the same search under four cost terms: none, parameter count, FLOPs, and measured latency. For each, we sweep the penalty weight to trace accuracy against true latency. We do this with plain DARTS and with two anti-collapse methods, DARTS- and Fair DARTS.
Part two: method, using an extended search space. Each 1×1 and 3×3 convolution in the space gets three execution modes. The objective is
$$\min_{\alpha,\beta}\; \mathcal{L}_{\text{val}}(w^*,\alpha,\beta) + \lambda \sum_{e}\sum_{o} p_e(o) \sum_{m} q_{e,o}(m)\, T_d(o,m).$$
The symbols are defined as follows:
- $p_e(o)$ is the probability of operation $o$ on edge $e$, set by architecture parameters $\alpha$.
- $q_{e,o}(m)$ is the probability of execution mode $m \in \{\text{dense}, 2{:}4, \text{unstructured}\}$, set by parameters $\beta$.
- $T_d(o,m)$ is the latency measured on device $d$ for that operation in that mode at that layer's tensor shape.
- $\lambda$ trades accuracy against speed.
The three modes of an operation apply different masks to one shared weight tensor $w$. Masks are recomputed from weight magnitude, and SR-STE is used for the sparse modes. Mode selection uses independent sigmoids rather than a softmax, following Fair DARTS, so that dense and sparse versions do not suppress each other.
The latency table $T_d$ is measured once per device on three targets:
- an Ampere-class GPU, which has hardware 2:4 support;
- the Jetson TX2 used in HW-NAS-Bench, which has no sparse tensor cores;
- a Raspberry Pi 4 CPU.
Unstructured sparsity is timed with the best kernels we can obtain, including those from Sparse GPU kernels for deep learning. That way the unstructured mode is not handicapped by a weak implementation.
Baselines
The naive reference point is DARTS with no efficiency term. It is followed by dense training and then uniform 2:4 sparsification of every convolution.
The strongest competitor is the decoupled pipeline. A stabilized dense search (DARTS- or TF-NAS) is penalized by measured latency so that the topology itself is fair. The resulting dense network is then sparsified, either with SR-STE fine-tuning (Zhou et al.) or with the proximal 2:4 operator (Kübler et al.). We also give this pipeline a cheap per-layer choice between dense and 2:4, made greedily against the same latency table. For reference, Zhou et al. report 77.0% ImageNet top-1 for 2:4 ResNet-50 against 77.3% dense. On the latency-search side, TF-NAS reports 76.9% top-1 with 1.8 GPU-days of search, and LA-DARTS reports 2.72% CIFAR-10 error at 28.4 ms against 40.9 ms for DARTS.
The black-box reference is regularized evolution on NAS-Bench-201 using measured latency. NAS-Bench-Suite ranks regularized evolution first among black-box optimizers, with an average rank of 2.36. The space optimum is 94.37% CIFAR-10 test accuracy (NAS-Bench-201). No reported result found on Lacuna for latency-penalized DARTS on NAS-Bench-201, so we run that baseline ourselves.
The ablations isolate the two ingredients. The first gives sparse modes their own separate weights instead of shared ones, which tests whether sharing prevents a new under-training bias. The second swaps $T_d$ for FLOPs while keeping everything else fixed.
Fairness rules. All methods get the same data splits and the same number of search epochs. Each gets an equal tuning budget over $\lambda$ (eight values each). Every discovered network is retrained with the same final recipe, and three search seeds are run per method. "Beating" a baseline means a higher accuracy-versus-measured-latency hypervolume on the target device, with non-overlapping seed intervals. On NAS-Bench-201, it additionally means finding architectures closer to the tabular optimum at equal latency.
Why not just…?
Why not just search a dense topology and prune it to 2:4 afterward? This is the real threat. On an Ampere GPU, if uniform 2:4 is nearly free in accuracy for every layer, the joint search may add nothing. The decoupled baseline above, including its per-layer greedy variant, is designed to settle this question. We expect a gain only where the best topology depends on which layers can be sparse. One example is a cell that would pick a wider 3×3 convolution if it knew that convolution could run as 2:4. If the hypervolumes match on Ampere and on TX2, we will report that sparsity can be decided after search, and the contribution shrinks to the diagnosis.
Why not just use an anti-collapse method with a FLOPs penalty? DARTS- or Fair DARTS might remove the bias regardless of the cost model. Part one tests exactly this by crossing the anti-collapse methods with the cost models. If the fraction of parameter-free operations and the accuracy at matched latency do not depend on the cost model once DARTS- is used, then our amplification claim is false for stabilized search.
Why not just run evolution with measured latency? On NAS-Bench-201 this is cheap and strong, so we include it as a reference. In the sparse-mode space no table exists, and every candidate would have to be trained with its masks. MODNAS reports that multi-objective black-box baselines need more than 370 hours, compared with 25.25 GPU hours for its supernet. We will report evolution under a matched training budget in the extended space.
Why not just use a learned latency predictor, as LA-DARTS does? A predictor is a valid replacement for our measured table. The difference is what it covers: LA-DARTS predicts dense execution only, and its paper does not report how its penalty affects collapse. We will compare a predictor against the measured table to check that our results do not depend on how latency is obtained.
Decisive Checks
Three outcomes decide the proposal.
- Collapse attribution. On NAS-Bench-201 at matched true latency, the FLOPs-penalized search should end with a larger share of parameter-free operations than the latency-penalized search. The difference should be larger on the Raspberry Pi and Edge TPU than on the Edge GPU. If the shares are equal, the amplification hypothesis fails.
- Device flip. The extended search should choose 2:4 modes on Ampere but mostly dense modes on TX2, which has no 2:4 hardware. It should rarely choose unstructured modes on either GPU. If the search picks unstructured sparsity on a GPU under measured cost, then either our latency table or our kernels are wrong.
- End-to-end ceiling. Measured end-to-end speedups from sparse modes should stay below the per-layer gains, roughly in line with the 1.21× Amdahl limit reported by Hu et al.. This confirms that the search is optimizing real wall-clock time and not an idealized per-layer figure.
References
- Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search
- DARTS-: Robustly Stepping out of Performance Collapse Without Indicators
- Understanding and Robustifying Differentiable Architecture Search
- Latency-Aware Differentiable Neural Architecture Search
- TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
- DEff-ARTS: differentiable efficient ARchiTecture search
- Multi-objective Differentiable Neural Architecture Search
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- Accelerating Transformer Pre-training with 2:4 Sparsity
- A Proximal Operator for Inducing 2:4-Sparsity
- Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
- NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search
- Sparse GPU Kernels for Deep Learning
- NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy