Training a network to be sparse usually means adding a penalty (a cost on how many weights or channels stay active) to the ordinary average training loss, scaled by a weight $\lambda$. Practitioners know this knob is treacherous. Make $\lambda$ slightly too small and nothing gets pruned. Make it slightly too large and the network cuts pathways it needed, and accuracy collapses. The good setting also does not carry over from one dataset or architecture to the next. We think part of the problem is the average loss itself. Average loss always rewards pushing already-easy examples toward zero loss. Its gradient therefore never settles at a natural stopping point, so the balance between "fit better" and "be sparser" slides continuously as $\lambda$ changes.
The proposal replaces the average loss with a two-sided threshold risk. This objective, due to Holland, penalizes each example's loss for straying from a target level $\theta$ in either direction, above or below. Under it, the fit term exerts almost no force once losses sit near $\theta$ and pushes back hard once pruning drives them above it. The sparsity level a run ends at should then be set mainly by $\theta$, a loss level that is easy to interpret and plausibly transferable, and hardly at all by $\lambda$ over a wide range. The claim is testable: the range of $\lambda$ values that produce a usable sparse model should widen by orders of magnitude, and a $\theta$ expressed relative to the dense model's training loss should transfer across datasets where $\lambda$ does not.
Core Claim
We attach standard sparsifiers to Holland's threshold risk and pin the target at a fixed $\theta$. The sparsifiers are the stochastic on/off gates of Learning Sparse Neural Networks through L_0 Regularization and the batch-norm scale penalty of Learning Efficient Convolutional Networks through Network Slimming. The threshold risk comes from Flexible risk design using bi-directional dispersion. The training objective is
$$\min_{w,\phi}\ \mathbb{E}_{(x,y)}\,\rho_\alpha\big(\ell(x,y;w,\phi)-\theta\big) \;+\; \lambda\,R(\phi).$$
Here $w$ are the network weights and $\phi$ the gate or channel-scale parameters. $\ell$ is the per-example cross-entropy and $R$ is the sparsity penalty, either the expected number of open gates or the $L_1$ norm of batch-norm scales. $\rho_\alpha$ is a member of the Barron family, a one-parameter set of penalties that runs from quadratic ($\alpha=2$) through Huber-like ($\alpha=1$) to bounded-gradient ($\alpha\le 0$).
At a stationary point, each gate parameter $\phi_j$ balances
$$\mathbb{E}\big[\rho_\alpha'(\ell-\theta)\,\partial_{\phi_j}\ell\big] \;=\; -\lambda\,\partial_{\phi_j}R.$$
Under ordinary average loss, $\rho'\equiv 1$. The left side is then the plain average gradient, and it stays nonzero while any example can still be fit better, so every change in $\lambda$ moves the equilibrium. Under the threshold risk, $\rho'(0)=0$. The fit force vanishes when losses hover at $\theta$ and grows roughly in proportion to $\rho''(0)$ times how far losses rise above it. A given $\lambda$ can therefore be balanced by only a small excess loss above $\theta$. The endpoint is pinned near $\theta$, and $\lambda$ mostly sets how fast the run gets there.
The same equation predicts where the stable window ends, and this prediction is falsifiable. For $\alpha\le 0$, $\rho'$ is bounded. That bound gives the "implicit gradient clipping" Holland reports, but it also caps how hard the fit term can resist. Above a critical $\lambda$ the gates should collapse regardless of $\theta$. So robust $\alpha$ buys stability under noisy labels at the price of a lower upper edge on $\lambda$, and we will measure this trade-off rather than assume it away.
Why Two-Sidedness, and Where It Might Not Matter
We have to be honest about one point. The pinning argument above only needs a flat-bottomed penalty. A one-sided hinge, which penalizes only losses above $\theta$ and resembles the upper-tail objectives Holland contrasts against, would pin the endpoint too. The lower half of the two-sided penalty adds something different: it pushes examples that are already below $\theta$ back up toward it.
That can cut two ways, and the file offers no evidence to pick between them. It may help. Capacity no longer goes into driving easy examples to zero loss, so when pruning bites, the damage spreads across examples instead of landing on the rare and hard ones. Holland's finding that these risks tighten the test-loss distribution in Learning with risks based on M-location points this way. It may also hurt. The network can satisfy the lower-tail pressure by becoming less confident rather than by pruning, which would yield less sparsity at the same $\theta$.
The off-policy work in Exponential Smoothing for Off-Policy Learning motivates taking two-sidedness seriously. It shows that controlling deviations in only one direction lets an objective under-report true risk, and that bounding both sides is what makes the criterion trustworthy to optimize. Whether that logic carries over to pruning is exactly what the one-sided ablation tests. The central contribution stands either way: it is the loss-target reparameterization of sparsity strength. The two-sided part is a hypothesis with a clean experiment attached.
Relation to Prior Work
Holland's two papers supply the objective, together with its weak-convexity convergence guarantee and its bounded-gradient behavior. Both evaluate only dense models and never touch pruning or penalty sensitivity. The sparsity papers supply the evidence for the gap. Louizos et al. and Liu et al. each report that their trade-offs hinge on $\lambda$. Soft Weight-Sharing for Neural Network Compression resorts to Bayesian optimization to set its accuracy–complexity balance, and A Sparse Deep Factorization Machine for Efficient CTR prediction finds the frontier strictly dataset-dependent.
Two works come closest, and each constrains the design.
Weight Pruning via Adaptive Sparsity Loss removes $\lambda$ by asking for a target sparsity. We instead ask for a target loss. That answers a different question, "how sparse can I go at this fit," and does not require knowing the right sparsity in advance.
Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity (WCR) reshapes the distribution of weight magnitudes for one-shot pruning after training, but it keeps average loss. We reshape the distribution of per-example losses during regularized training. The two act on different objects and could in principle be combined, and we test that combination.
Research Program
The build is small by design. We implement the threshold risk as a drop-in replacement for cross-entropy inside two existing sparsifiers: Hard Concrete $L_0$ gates, where each unit has a learned probability of being switched off, and Network Slimming. We express $\theta$ as $c$ times the dense model's final training loss, with $c$ a dimensionless multiplier.
The analysis has three parts. The first is empirical sensitivity maps: final sparsity and accuracy as functions of $(\lambda, \theta, \alpha)$. The second is a check of the stationarity prediction. During training we log the fit-term force on the gates and the gap between the mean loss and $\theta$. The prediction is that the gap scales like $\lambda/\rho''(0)$ inside the window, and that collapse sets in near the $\lambda$ at which the bounded $\rho'$ saturates when $\alpha\le 0$. The third is a transfer study: a $c$ chosen on one dataset and architecture, applied unchanged to others.
The central sensitivity measure is the usable $\lambda$ window. For each method we sweep $\lambda$ over six decades in half-decade steps with three seeds. The window is the number of decades over which the run keeps at least a fixed sparsity (80% of gates or channels removed) while staying within one accuracy point of that method's best sparse run. We also report the slope of final sparsity against $\log\lambda$, the variance of the endpoint across seeds, and the upper-tail loss and worst-class accuracy of the pruned model. The file contains no prior measurement of such a window for any method, so every number here is new.
Baselines
The naive reference point is the standard recipe: average cross-entropy plus the sparsity penalty, run with the published training schedules of Louizos et al. and Liu et al. and swept over the same six-decade $\lambda$ grid. This recipe produced the reported results, for example a 3.83% error for $L_0$-gated Wide ResNet on CIFAR-10 in Learning Sparse Neural Networks through L_0 Regularization and 5.19% error for DenseNet-40 at 40% channel pruning in Learning Efficient Convolutional Networks through Network Slimming. Beating it means two things. The window must be at least two decades wider. And the best-in-sweep accuracy must not be worse by more than seed noise, so that we are not buying stability with a worse optimum.
The strongest existing competitor for removing $\lambda$ is the adaptive sparsity loss of Retsinas et al. It reports 78.66% accuracy on CIFAR-100 with Wide ResNet-16-8 pruned to 1.6M parameters, against a 78.52% dense baseline (Weight Pruning via Adaptive Sparsity Loss). It is strongest because it already removes the penalty weight. To compare fairly, we run our method at a given $c$, record the sparsity it reaches, and hand that exact sparsity to Retsinas' method as its budget, with the same epochs and augmentation. We then compare accuracy and upper-tail loss. Matching within 0.3 points while also requiring no prior knowledge of the target sparsity counts as a win. Clearly exceeding it on worst-class accuracy would count as a stronger one.
The strongest constrained-optimization competitor minimizes the penalty subject to a mean training-loss constraint at $\theta$, with $\lambda$ updated by dual ascent. This is the textbook way to "target a loss." We tune its dual step size on the same grid budget we give $c$.
For one-shot pruning after training, the reference is WCR with sharpness-aware minimization (SAM). At 96% pruning of ResNet-18 on CIFAR-10 it reports 92.08%, against 48.36% for SAM alone (Weight Concentration Regularization). We evaluate it at matched sparsity as a separate comparison, since it addresses a different pipeline.
The key ablation swaps the two-sided $\rho_\alpha$ for a one-sided hinge with the same curvature above $\theta$. This isolates what the lower-tail term contributes. Every method gets the same number of training runs for its own knobs: $\lambda$, $c$, dual step size, or budget.
Why not just…?
Why not just grid-search or Bayesian-optimize $\lambda$? For a single dataset this works, given enough compute. Ullrich et al. needed Bayesian optimization exactly because the balance was fragile, and search does nothing to make the result transferable. Our claim is not that $\theta$ needs no tuning. It is that $c$ tuned once transfers while $\lambda$ does not. The transfer experiment tests this directly. If a $\lambda$ chosen on CIFAR-10 transfers as well as $c$ does, the proposal's practical value largely disappears, and we will report that.
Why not just specify a sparsity budget, as the adaptive sparsity loss does? Budget methods are the right tool when you know the sparsity you want. They say nothing about where accuracy begins to break, which is usually the thing practitioners are searching for. The matched-sparsity comparison checks whether our route to a given sparsity costs any accuracy relative to Retsinas et al. If it does, the loss-target view survives only as a way to choose the budget, and not as a training method.
Why not just enforce a mean-loss constraint with a learned multiplier? This is the most serious threat, because it produces the same pinning effect as our stationarity equation and needs no dispersion function. It differs in two ways. It adds a dual step size that can itself oscillate, and it constrains only the mean, leaving the per-example spread free. If dual ascent matches us on window width, transfer, and upper-tail loss, then the honest conclusion is that loss targeting, not bidirectional dispersion, is the operative mechanism. The experiment is designed to reveal exactly that.
Why not just clip gradients or use SAM? Holland argues that explicit clipping adds its own threshold, and Yun et al. show SAM alone collapses at high sparsity: 9.74% on SVHN at 96% pruning. Neither alters the equilibrium the penalty is balanced against, which is where we locate the instability. We include clipped-gradient average loss in the $\lambda$ sweep at a single well-tuned clip value to confirm this.
Proposed Experiments
The main sweep runs $L_0$-gated LeNet-5 on MNIST and Wide ResNet on CIFAR-10, Network Slimming on VGG and DenseNet-40 for CIFAR-10, and Wide ResNet-16-8 on CIFAR-100. It crosses six decades of $\lambda$ with four values of $c$ and $\alpha\in\{2,1,0,-2\}$, over three seeds, and compares against every baseline above. A label-noise variant of CIFAR-10, the setting where Holland found two-sided risks helped dense models, tests whether bounded-gradient $\alpha$ keeps the window wide when noisy examples would otherwise dominate the fit force.
The transfer check picks $c$ and $\lambda$ on CIFAR-10 with VGG and applies both unchanged to DenseNet-40, to CIFAR-100 with Wide ResNet, and to SVHN. It asks which of the two keeps the run inside the usable window.
Three outcomes would decide the question. First, whether the usable window widens by at least two decades over average loss at equal best accuracy. Second, whether the measured gap between mean loss and $\theta$ scales with $\lambda$ as the stationarity argument predicts, including a collapse point that moves with $\alpha$. Third, whether the two-sided penalty beats both the one-sided hinge and dual ascent on worst-class accuracy at matched sparsity.
A null on the first outcome refutes the proposal. A null on the third alone narrows it to a result about loss targeting rather than bidirectionality.
References
- Learning Sparse Neural Networks through L_0 Regularization
- Learning Efficient Convolutional Networks through Network Slimming
- Flexible risk design using bi-directional dispersion
- Learning with risks based on M-location
- Exponential Smoothing for Off-Policy Learning
- Soft Weight-Sharing for Neural Network Compression
- DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
- Weight Pruning via Adaptive Sparsity Loss
- Weight Concentration Regularization for Improving Pruning Robustness Under High Sparsity