Pruning can delete 80–95% of a network's weights while barely hurting accuracy. On a GPU, though, the pruned model often runs no faster than the dense one. The problem is where the surviving weights sit. A fast sparse matrix kernel on Tensor Cores (the GPU's dense matrix-multiply units) packs the nonzeros into small dense blocks. Its speed then depends on how many blocks are occupied and how evenly they spread across rows, not on how many weights survive. Today's pruning methods never see this. They choose weights by magnitude or gradient and leave the kernel to cope with whatever pattern results. This proposal treats kernel wall-clock time the way speech enhancement treats PESQ (a perceptual quality score): as a metric that has no derivative but can be measured. It then borrows that field's hard-won recipe for training against such metrics. A small differentiable model of latency guides the pruning mask. Real kernel timings periodically retrain that model. The whole loop is audited for the optimizer learning to fool its own proxy.
In two sentences: we attach a cheap, differentiable latency predictor to a soft-mask sparse trainer and refit it in a closed loop against real GPU timings, so the mask is pushed toward patterns the kernel actually runs fast. We test whether this beats both rigid hardware-friendly sparsity patterns and "prune first, reorder later."
Core Claim
A latency surrogate for unstructured sparsity only works if it has two properties. First, it must be written in terms of the quantities the kernel pays for: occupied blocks after row reordering, and the heaviest row's load. Second, it must stay calibrated as the mask moves. That requires the alternating re-measurement and replay machinery that metric-surrogate training developed. Without it, the optimizer drifts into mask regions where the predictor is wrong. We claim this closed-loop surrogate gives real, measured speedups at matched accuracy that fixed analytical cost models cannot deliver, and that post-hoc kernel reordering of ordinarily pruned networks cannot deliver either.
Relation to Prior Work
Four papers shape the proposal, each in a different way.
End-to-End Neural Network Compression via 𝓁1/𝓁2 Regularized Latency Surrogates builds its differentiable latency term by linearly interpolating per-layer latency lookup tables. It covers channel pruning, rank and bit-width only, and names block sparsity as future work. It is the closest existing work, and we differ in both what the surrogate sees (the mask's block structure) and how it stays accurate (live re-measurement instead of a fixed table).
Spartan supplies our mask mechanism: a soft top-k mask computed by entropic optimal transport, with a sharpness knob annealed from soft to hard. It reaches 76.5% ImageNet top-1 on ResNet-50 at 95% sparsity. Its cost term, however, is linear per weight, it reports no wall-clock timings, and it admits that nonlinear hardware overheads are out of reach. We keep its mask and replace the linear cost with a nonlinear, measured one.
HALP motivates the nonlinear shape of the cost. It documents that GPU latency falls in discrete "staircase" steps and exploits those steps with an offline knapsack solver over channel groups. That solver is not differentiable, and it would face an exponentially larger search space at weight granularity.
MAPLE-Edge constrains the design. It shows that layer-wise lookup tables mispredict latency once a runtime fuses layers, and it fixes this with a learned predictor that it only ever uses for evaluation, never as a source of gradients.
The kernel we target is SMaT from High Performance Unstructured SpMM Computation Using Tensor Cores. It packs unstructured matrices into blocked compressed sparse row format (small dense tiles of FP16 values), after Jaccard-similarity row reordering. It beats dense cuBLAS on A100 at sparsity as low as 78%, but it notes that uneven rows still create straggler warps that slow the whole kernel.
The stabilization recipe comes from metric surrogacy. MetricGAN+ alternates between refitting its quality predictor on fresh measurements and updating the generator, and keeps a replay buffer of past samples; it reaches PESQ 3.15 against 2.71 for plain MSE training. The PESQetarian is the cautionary result: optimizing hard against a PESQ proxy yields PESQ 3.82 alongside SI-SDR of −19.8 dB, a model that games the score rather than improving audio. We treat that as a direct prediction for our setting, where a mask could look fast to the proxy while running slowly on the kernel, and we build the audit to catch it.
Research Program
The mask comes from Spartan's soft top-k operator with sharpness $\beta$. For each weight matrix we hold a row permutation fixed within a phase and recompute it by Jaccard reordering only at re-measurement points, because reordering itself is not differentiable. Within a phase, we tile the permuted matrix into the kernel's blocks. The soft occupancy of block $b$ is $o_b = 1 - \prod_{i \in b}(1 - m_i)$, where $m_i \in [0,1]$ is the soft mask value of weight $i$. This quantity is 1 if any weight in the block survives, and it falls smoothly toward 0 as all of them are pruned.
From these occupancies we compute two features: total occupied blocks, and a log-sum-exp approximation of the maximum blocks in any row, which stands in for the straggler warp. A small per-layer model $T_\phi$ takes these features and the layer shape and predicts kernel time. It has two parts: a linear-in-features term with learned coefficients, plus a small neural residual that captures staircase and alignment effects. Training minimizes
$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \, \max\!\Big(0, \sum_\ell T_\phi(o^{(\ell)}) - T_{\text{budget}}\Big),$$
where $\ell$ indexes layers, $T_{\text{budget}}$ is the target end-to-end latency, and $\lambda$ weights the penalty.
Every $K$ steps the loop re-measures. It hardens the current mask, runs the kernel's preprocessing, times each layer and the full network on the GPU, adds the (features, latency) pairs to a replay buffer, and refits $\phi$ on the buffer. This is MetricGAN+'s alternating scheme with latency in place of PESQ. The buffer is seeded before training with random and magnitude-pruned masks, so the proxy has seen poor configurations as well as good ones.
The Goodhart audit logs, at every re-measurement, the gap between predicted and measured latency on the current mask. This gap is exactly the error the optimizer has been exploiting since the last refit.
Baselines
The naive reference is a dense model run with cuBLAS, together with the same network pruned to matching sparsity by RigL, then executed with SMaT including its reordering step. RigL reports 74.6% ImageNet top-1 on ResNet-50 at 80% sparsity. This baseline answers the question "does the kernel alone already rescue ordinarily pruned networks?"
The strongest accuracy-side competitor is Spartan with its linear cost vector, also run through SMaT. Its accuracy figure is 76.5% top-1 at 95% sparsity. There is no reported result found on Lacuna for Spartan's measured wall-clock latency, so we will measure it ourselves.
The strongest hardware-side competitors are the rigid patterns. For 2:4 sparsity trained with SR-STE, Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch reports 77.0% top-1 against a 77.3% dense baseline. For tile-wise sparsity, Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity reports an average 1.95× Tensor Core speedup on V100 with 1–3% accuracy loss at 75% sparsity.
Two ablations isolate the proposal's ingredients. The first uses the analytical block-occupancy penalty with hand-set coefficients: no learned model and no re-measurement. The second fits $T_\phi$ once before training and then freezes it.
Every method trains on ImageNet-1K for ResNet-50, and on the GLUE benchmark for BERT, using the same epoch budget, the same number of hyperparameter trials, and the same GPU. Latency is always measured on hardened masks with SMaT preprocessing included. We claim to beat the baselines if we reach a lower measured end-to-end latency at equal top-1 (within 0.3 points) than both Spartan-plus-SMaT and tile-wise sparsity on A100. On V100, the claim is that we match or exceed tile-wise sparsity's roughly 1.95× speedup with smaller accuracy loss.
Why not just…?
Why not just use 2:4 or tile-wise sparsity? This is the real threat. Tile-wise sparsity already delivers about 2× on Tensor Cores with no special hardware, and 2:4 costs only 0.3 top-1 points. The counterarguments are budget and allocation. 2:4 caps sparsity at 50% and forces the same ratio everywhere, while RigL and dynamic sparse reparameterization show that good networks want uneven density across layers. Tile-wise sparsity loses 1–3 points at 75%. If our method cannot beat tile-wise sparsity on the accuracy–latency frontier at 80–95% sparsity, the proposal fails, and we will report that outcome.
Why not just write down block occupancy analytically and penalize it? This could plausibly match the learned proxy for isolated kernels, where time may be close to linear in block count. That is why the design keeps an analytical backbone, and why the first ablation tests the analytical penalty alone. The learned residual and the re-measurement loop earn their place only if two things happen: measured end-to-end latency, including fusion and staircase effects, departs from block counts; and the analytical-only run shows a growing gap between predicted and measured latency. If neither happens, the simpler penalty is the recommended result.
Why not just prune normally and let SMaT's reordering fix the layout? SMaT itself notes that reordering cannot remove straggler rows under highly skewed nonzero distributions. The RigL-plus-SMaT and Spartan-plus-SMaT baselines measure directly how much speed post-hoc reordering recovers, and our method must beat both at matched accuracy.
Why not just search over masks offline, HALP-style? At weight granularity the search space grows as $2^{|W|}$, where $|W|$ is the number of weights. HALP's solver is tractable only because it chooses among channel groups.
Decisive Checks
The central experiment is the measured latency–accuracy frontier on A100 with SMaT, and on V100 as the portability check, comparing all baselines at 80%, 90% and 95% sparsity.
The second check tests the Goodhart prediction. We plot the proxy-versus-measured gap against the refit interval $K$. The frozen-proxy ablation should show the gap widening over training. The closed-loop version should keep it bounded. If that contrast does not appear, the metric-surrogacy machinery is unnecessary in this setting.
The third check reports training cost, meaning measurement time as a fraction of total training time, and compares it with the roughly 0.5% overhead that Differentiable Transportation Pruning reports for its optimal-transport step, to show whether the loop is affordable.
References
- End-to-End Neural Network Compression via l1/l2 Regularized Latency Surrogates
- Spartan: Differentiable Sparsity via Regularized Transportation
- HALP: Hardware-Aware Latency Pruning
- MAPLE-Edge: A Runtime Latency Predictor for Edge Devices
- High Performance Unstructured SpMM Computation Using Tensor Cores
- MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement
- The PESQetarian: On the Relevance of Goodhart's Law for Speech Enhancement
- Rigging the Lottery: Making All Tickets Winners
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- Accelerating sparse DNN models without hardware-support via tile-wise sparsity
- Parameter efficient training of deep convolutional neural networks by dynamic sparse reparameterization
- Differentiable Transportation Pruning