Unstructured weight pruning deletes individual weights anywhere in a matrix. It can remove most of a network's arithmetic and still leave inference no faster on GPUs and phone CPUs, because those chips need regular, contiguous memory access to run in parallel. The fix that commodity hardware actually rewards is structured fine-grained sparsity. One form is 2:4 sparsity, where exactly two of every four consecutive weights are zero, which NVIDIA Ampere Tensor Cores can skip. Another is SIMD-structured pruning, which removes weights in groups of four channels so the pruned layout matches a 128-bit CPU vector register. Separately, hardware-aware architecture search trains one large "supernet" whose sub-networks share weights, so that many candidate models for many devices can be pulled from a single training run. These two lines have not met. Supernets vary width, depth, and bit-width, but every weight matrix inside them is dense. The one supernet that does vary sparsity uses unstructured masks and scores efficiency by model size rather than measured latency.
We propose a supernet in which the per-layer choice of hardware-aligned sparsity pattern is an architectural dimension alongside width and depth. It is trained so that every sub-network's mask is a nested subset of a single shared weight tensor, with channel widths that snap to hardware tile boundaries. The central question is not whether such a supernet can be built. It is whether choosing sparsity jointly with macro-architecture finds latency–accuracy trade-offs that the obvious sequential pipeline cannot. That pipeline searches a dense architecture first and then sparsifies it, uniformly or with a latency-aware per-layer allocator. If joint search wins, the reason should be visible: under measured latency, a wide 2:4-sparse layer can be cheaper than a narrow dense one, so the best topology itself should shift.
Core Claim
Weight sharing only works if the thing being shared matches the thing hardware executes. In an elastic-width supernet, a sub-network takes a prefix of each layer's channels. In a 2:4 or SIMD-structured model, the unit of execution is a block of four weights or four channels. We claim that when these units are aligned, three things follow. Widths are restricted to multiples of the block and tile size. Masks at different densities are nested, derived from one importance ordering. Any channel reordering is confined to within tile segments so that prefixes keep the same members. Under these conditions a single supernet can serve many sparsity levels and devices without the mask conflicts that would otherwise corrupt shared weights. We also claim that the resulting joint search beats dense-search-then-sparsify at matched measured latency on an Ampere GPU and an ARM Cortex-A72 CPU.
This is distinct from the closest existing work. Omni-sparsity DNN trains one supernet over 50–80% unstructured sparsity for speech recognition, using sandwich sampling (training the largest, smallest, and random sub-networks in each batch). It reports 3x lower training cost and 2% to 6.6% better WER than individually pruned models, but it never measures latency and its masks have no hardware structure. HAT and Single Path One-Shot (SPOS) search width, depth, and bit-width under latency constraints, but with dense weights and channel choices not grouped by hardware steps. We take the nested sparsity training idea from the first and the measured-latency evolutionary search from the second. We add the constraint, missing from both, that every choice lands on a boundary the hardware rewards.
Research Program
Shared tensor and nested masks. For each prunable layer we keep one weight tensor and one learned importance score per weight or per SIMD group. Weights are grouped into blocks of $M$ along the dimension the kernel requires. For a density $d$, the mask keeps the top $\lceil dM \rceil$ entries of each block by score. This gives masks that are nested by construction:
$$\text{mask}(d_1) \subseteq \text{mask}(d_2) \quad \text{whenever } d_1 < d_2 .$$
Nesting is what lets sub-networks at different densities share weights without fighting over which weights exist.
Target-specific density choices. The set of densities is chosen per target and only includes patterns the hardware actually accelerates. On Ampere GPUs the per-layer choice is {dense, 2:4}, since other N:M ratios get no native speedup there. On the Cortex-A72 the choices are several SIMD-group densities, following the density grid of Architecture Aware Latency Constrained Sparse Neural Networks (ALCS).
Training the masked weights. To keep masks from flickering under sampling, we reuse the sparse-refined straight-through estimator (SR-STE) of Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch, a gradient trick that lets masked weights keep learning so masks stabilize. That paper trains one uniform-2:4 model; we apply the estimator per sampled sub-network. We also track its sparse-architecture-divergence measure of mask churn as a training diagnostic.
Tile-aligned widths. Channel widths are offered only at multiples of the GPU tile (8 or 16) and the SIMD width (4). This builds in the latency "staircase" that HALP documents, where pruning within a tile step saves FLOPs but no time. HALP imposes this on post-hoc filter pruning of one model; we impose it on the search space itself.
Segment-constrained permutation. Channel Permutations for N:M Sparsity shows that reordering channels before 2:4 pruning rescues compact models: SqueezeNet goes from a 4.01% drop to a 0.60% gain. The trick relies on the previous layer's output order matching the next layer's input order. A supernet complicates this, because the previous layer is sampled at many widths. We resolve it by permuting only inside each width-tile segment. The set of channels in every allowed prefix is then unchanged, and each sub-network inherits a valid, overhead-free reordering. Whether this restricted permutation keeps most of the accuracy benefit of an unrestricted one is itself a measured question.
Latency prediction. Latency comes from a predictor trained on whole sub-networks measured with the deployed runtime: TensorRT on the GPU, and the compiled sparse SIMD kernels on the ARM board. We do not use an additive per-layer lookup table. HW-NAS-Bench notes that such tables ignore operator fusion, and the speedup a sparse kernel gives depends on layer shape and on what surrounds it.
Search. Search is evolutionary under a latency budget, as in SPOS. We optionally add the listwise Pareto ranking loss of Pareto Rank-Preserving Supernetwork, which trains the supernet to preserve which sub-networks dominate which. That work covers dense operation choice; we test whether it transfers once a sparsity dimension is added.
Baselines
Naive reference. The do-nothing reference is the dense sub-network found by our own supernet with the sparsity dimension removed, searched under the same measured-latency predictor and fine-tuned identically. Every gain from sparsity is reported against it.
Uniform 2:4 after dense search. The first sequential baseline takes that dense searched network and applies uniform 2:4 sparsity with SR-STE. SR-STE is strong: the source paper reports 77.0% ImageNet top-1 for ResNet-50 at 2:4 against 77.3% dense. We will run it with the same fine-tuning epochs we give our extracted sub-networks, and with channel permutation added, so that we are not beating a weakened version.
Latency-aware per-layer allocation after dense search (strongest competitor). The second sequential baseline, and the one we consider strongest, is a HALP-style knapsack allocator. It uses measured latency and importance scores to decide, layer by layer, which layers of the dense searched network become 2:4 or which SIMD density each layer gets. On ARM we also run ALCS's ADMM pruning on the same network. HALP reports 1.60x throughput with +0.3% top-1 for ResNet-50 on ImageNet. ALCS reports 2.68x on ResNet-18 and 2.04x on MobileNetV1 on a Cortex-A72 with no accuracy loss. These are the per-model speedup levels a sparsified network must reach. This baseline captures everything except the architecture search responding to sparsity, which is exactly what our claim rests on. All baselines receive the same total GPU budget, including per-target retraining, and the same latency measurement protocol. Beating this baseline means a higher Pareto hypervolume (the area of the accuracy–latency trade-off curve covered by a method's models) over the same latency range on each device, with at least one budget where top-1 accuracy is higher by more than seed-to-seed variation.
Dense hardware-aware supernets. For external context we report dense SPOS-style search, with SPOS itself reporting 74.7% top-1 at about 300M FLOPs. We also report the Pareto rank-preserving supernet, which reports 77.2% at 3.68 ms on an edge GPU and a Kendall rank correlation rising from 0.47 to 0.97 on NAS-Bench-201. We re-measure both on our devices, since their latencies come from other hardware.
Ablations. Two ablations isolate the key ingredients. One replaces nested masks with independent per-density masks. The other replaces segment-constrained permutation with none, and separately with unrestricted permutation computed for the largest sub-network only.
Why not just…?
Why not just search a dense architecture and then allocate 2:4 per layer with a latency-aware knapsack? This is a genuine threat, and the proposal is built to settle it. If sparsity cost and topology were separable, the sequential pipeline would match joint search at lower complexity. The test is direct. We compare Pareto fronts, and we check whether jointly searched architectures differ systematically from dense-searched ones, for example by being wider in layers that become 2:4. If the fronts coincide, we will report that measured-latency sparsity is separable from macro-search, which is itself useful to practitioners. The supernet's remaining advantage would then be amortization across devices and budgets, the 3x training saving Omni-sparsity reports, and we would quantify it rather than claim it.
Why not just apply uniform 2:4 everywhere with SR-STE? On large ResNets this nearly suffices. Compact mobile models are the regime where uniform 2:4 hurts, as the 4.01% SqueezeNet drop shows, and they are also the regime edge search targets. Uniform 2:4 also does nothing on ARM CPUs. We include it as a baseline on both devices so its shortfall is measured rather than assumed.
Why not just penalize FLOPs or sparsity percentage in the supernet objective, as Omni-sparsity does? HW-NAS-Bench reports FLOPs–latency rank correlation as low as 0.18 on an Edge TPU and below 0.5 on several devices. A 2:4 layer halves FLOPs but may not halve time for small shapes. We will run our supernet with a FLOPs objective instead of measured latency, and report how many of its chosen models fall off the measured Pareto front.
Why not just keep unstructured sparsity and use fast custom kernels? Custom-kernel approaches such as HASS (a dedicated FPGA engine) or reordering libraries like SMaT need hardware or runtimes outside standard deployment stacks, and generic sparse libraries slow down at 70–95% sparsity. Our scope is commodity silicon with stock runtimes. We state that restriction openly rather than claiming superiority over custom accelerators.
Proposed Experiments
Main comparison. The search space is MobileNet-style (the FBNet layer space from HW-NAS-Bench), extended with tile-aligned widths and per-layer sparsity choices, trained on ImageNet-1K. Targets are an Ampere-class GPU using 2:4 through TensorRT and a Raspberry Pi 4 Cortex-A72 using SIMD-group sparsity. Each device's Pareto front compares our method against all baselines at five latency budgets, with three seeds, measured end to end on device.
Rank-correlation check. The first decisive check addresses the concern in NAS-Bench-1Shot1, which found near-zero correlation (Spearman between -0.25 and 0.3) between a sub-network's supernet accuracy and its accuracy when trained alone. On CIFAR-10 we sample 100 sub-networks spanning sparsity choices, train each standalone, and measure Kendall correlation with supernet-inherited accuracy. We do this with and without nested masks and with and without the Pareto ranking loss. If adding the sparsity dimension drops correlation well below the dense supernet's, search results are untrustworthy and nested masking has failed.
Permutation check. The second decisive check asks whether segment-constrained permutation recovers most of the accuracy that unrestricted permutation gives standalone 2:4 compact models.
Architecture-shift check. The third decisive check asks whether the jointly searched architectures differ structurally from dense-searched ones and outperform the knapsack-sparsified dense network at matched measured latency. Joint search being worth its complexity depends on this check.
References
- Omni-Sparsity DNN: Fast Sparsity Optimization for On-Device Streaming E2E ASR Via Supernet
- HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Architecture Aware Latency Constrained Sparse Neural Networks
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- HALP: Hardware-Aware Latency Pruning
- Channel Permutations for N:M Sparsity
- HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
- Pareto Rank-Preserving Supernetwork for HW-NAS
- HASS: Hardware-Aware Sparsity Search for Dataflow DNN Accelerator
- High Performance Unstructured SpMM Computation Using Tensor Cores
- NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search