A pruned network, meaning a large network with most of its weights removed, is usually sold as an efficiency win. It is cheaper to serve, and theory says it can even generalize better than a small network trained directly. That claim leaves out the cost of finding it. Under noisy labels, test accuracy does not fall smoothly as weights are removed. It dips, recovers, and only then collapses, a pattern called sparse double descent. Nobody can say in advance where the good region lies, so practitioners train and retrain at many sparsity levels to locate it. That search is exactly the kind of hidden development compute that full-lifecycle audits of AI find to be about half of training emissions. The field then counts only the final model's floating-point operations (FLOPs), ignores the search, and ignores that cheaper serving tends to attract more queries. This is the rebound effect, or Jevons' paradox: a per-unit efficiency gain partly or fully cancelled by increased use.
This note proposes two things. First, a lifecycle energy account of compression under double descent, measured in joules rather than FLOPs and covering every search, teacher, and deployment cost. Second, a cheap predictor that locates the good sparsity region from the dense model alone, so the search can be skipped. The central question is concrete. Once search, teacher training, and demand response are counted, when does "train big, then prune" save energy compared with training a small model directly? And does removing the search change the answer?
Core Claim
The overparameterization paradox has an energy consequence that has not been measured. Non-monotonic generalization makes the development phase expensive, because a practitioner cannot interpolate between a few trial points and must scan instead.
Provable Benefits of Overparameterization in Model Compression motivates the proposal. It proves that pruning an overparameterized model beats training a small model from scratch, even an "oracle" small model trained only on the informative features. Its framework, however, treats the large pre-pruning model as free.
Sparse Double Descent supplies the phenomenon. Under label noise, accuracy against sparsity is W-shaped. There is a "critical" band at moderate sparsity where test accuracy crashes while training accuracy stays near 100%. It is followed by a "sweet" band where the network is too small to memorize the noise but still large enough to fit the signal.
Holistically Evaluating the Environmental Impact of Creating Language Models supplies the accounting stance. Development runs for OLMo emitted 159 tCO2eq against 312 tCO2eq for the final runs.
The claim is that the energy advantage of pruning depends on two things. One is how much sweep is needed to land in the sweet band. The other is how strongly demand responds to cheaper inference. A predictor that finds the sweet band without a sweep is the ingredient that could turn a paper efficiency gain into a real one. The proposal also states the conditions under which even that fails.
Accounting Model
We compare two routes to a deployable model at a fixed target accuracy.
The first route is "small from scratch." Its total energy is $E_s = S_s + T_s + Q_s c_s$. Here $S_s$ is the energy of the hyperparameter search, $T_s$ is the energy of the final training run, $c_s$ is the measured energy per served query, and $Q_s$ is the number of queries served over the deployment period.
The second route is "large, then pruned." Its total energy $E_p$ has the same form but adds the dense training run and any teacher model to the search term. For both routes, demand responds to per-query cost as
$$Q(c) = Q_0 \left(\frac{c}{c_0}\right)^{-\varepsilon}.$$
In this equation, $Q_0$ is the query volume at a reference cost $c_0$, and $\varepsilon \ge 0$ is the demand elasticity. $\varepsilon = 0$ means no rebound, and $\varepsilon \ge 1$ means cheaper serving increases total serving energy (backfire).
The quantity we report is the breakeven query volume $Q^*$ at which $E_p = E_s$, as a function of $\varepsilon$. The functional form is the standard constant-elasticity demand used in rebound analysis. We do not estimate $\varepsilon$ from data, because nothing in the available traces identifies it. We therefore treat it as a scenario axis and report the elasticity at which each method's advantage disappears.
This extends the OLMo breakeven analysis of Morrison et al., who found that an OLMo 7B model needs 1 to 20 billion requests for inference to match training emissions. Their analysis compares training against inference for a single model. Ours compares two ways of producing a compact model and includes the search.
Research Program
The program has three parts.
Measurement. We rerun the sparse double descent setting with a GPU power meter active for every run, including discarded ones. The setting is ResNet-style networks on CIFAR-10, CIFAR-100, and Tiny ImageNet, with 0%, 20%, and 40% symmetric label noise, pruned by iterative magnitude pruning with rewinding as in He et al. Measuring in joules follows Energy Considerations of Large Language Model Inference and Efficiency Optimizations. That paper shows FLOPs and energy come apart: the software stack alone changes energy by up to 73% at identical FLOPs, and nominally cheaper architectures such as mixture-of-experts can cost up to 54% more energy.
Prediction. The predictor uses the sweet-band mechanism described in the sparse double descent literature: the band begins where the network can no longer memorize the mislabeled examples. It proceeds in three steps.
First, we estimate the number of examples the dense model is memorizing rather than learning. This uses a loss-trajectory signal that costs nothing extra to collect: examples fitted late in training and with high early loss.
Second, from two or three short runs of small sparse networks, we estimate how many examples a network at a given sparsity can fit. This is the effective model complexity of Deep Double Descent, the largest sample size a training procedure can drive to near-zero training error. We extrapolate that capacity across sparsity.
Third, we choose the sparsity at which predicted capacity falls just below the total sample count but stays above the estimated number of clean examples.
This is a heuristic, and Nakkiran et al. note that effective model complexity depends on an arbitrary error threshold. The experiments therefore test the predictor against the full sweep instead of assuming it works.
Deployment. We serve every final model and replay request arrival times from BurstGPT, about 10.31 million Azure OpenAI traces. We use its bursty inter-arrival pattern rather than steady load, because steady-state benchmarks misstate serving behavior. This gives measured per-query energy under realistic load, which then enters the $Q^*$ computation.
Baselines
Naive reference: small from scratch. A small network is trained directly at the pruned model's parameter count and given the same hyperparameter tuning budget. It is the obvious "don't waste compute" choice. Its search, training, and serving energy are all measured. To beat it, the pruned route must reach higher test accuracy at equal total measured energy, or equal accuracy at lower total energy, for query volumes below a stated $Q^*$.
Full-sweep iterative magnitude pruning. We follow Sparse Double Descent, scanning sparsity until the sweet band is found. This is the accuracy ceiling for pruning without distillation. It is run on identical data and architectures, and its entire sweep is charged. The predictor beats it if it reaches within one point of the sweep's best accuracy at a fraction of the sweep's energy.
Strongest existing method: DSD². Dodging the Sparse Double Descent distills from a sparse teacher and stops training early using an activation-entropy signal. It reports 86.89% accuracy on CIFAR-10 with 20% label noise at high sparsity, against 74.52% for vanilla pruning, and about 25% fewer training FLOPs. We reproduce it on our architectures. We charge it for the teacher's training and pruning sweep, which that paper does not count. Beating it means matching its accuracy within one point at lower total measured energy, or showing its reported FLOP saving disappears once the teacher is charged.
Ablation of the predictor's key ingredient. We keep the predictor but replace the estimated count of memorized examples with the true injected noise rate as an oracle, and separately with a noise count of zero. This separates the value of estimating memorization from the capacity extrapolation. We also run a fixed "standard" sparsity of 90%, the choice a practitioner makes without thinking.
Why not just…?
Why not just train a compact model from scratch? This is the most serious threat. The proof in Provable Benefits of Overparameterization in Model Compression favors pruning only at equal sample size, not at equal energy, so its result does not settle the question. If the small-from-scratch baseline wins at matched energy for plausible query volumes, that is the headline finding, and we will report it as such. The matched-energy comparison above is designed to decide this directly.
Why not just use DSD²? DSD² removes the accuracy dip, but it moves the search into training the teacher. Its 25% saving is stated in FLOPs, which Fernandez et al. show can misstate energy. Charging the teacher and measuring joules will show whether it still wins. If it does, the predictor's role narrows to choosing the teacher's sparsity without a sweep, which is still useful.
Why not just add weight decay or early stopping so the curve becomes monotonic? Tuning that regularization strength is itself a sweep, and Deep Double Descent notes the optimal amount is task-dependent. In the high-dimensional linear setting, Cheng et al. show that stopping short of interpolation is suboptimal. We include a tuned weight-decay arm with its tuning energy charged.
Why not just count FLOPs? We will report both FLOPs and joules. We will count how often the ranking of methods flips between the two. A method ranking that never flips would weaken the measurement component of this proposal.
Proposed Experiments and Decisive Checks
The first check is localization. Across 3 datasets, 3 noise rates, and 3 widths, we record how often the predicted sparsity falls inside the empirically found sweet band, and how much energy it costs relative to the sweep. Sparse Double Descent reports that the critical band is narrow or undetectable on clean data. We therefore expect little gain at 0% noise and say so in advance.
The second check is the matched-energy accuracy comparison of all routes, as described in the Baselines section.
The third check is the $Q^*$ curves under the replayed BurstGPT serving load, reported across a range of $\varepsilon$ from 0 to 1.5. For each method, we identify the elasticity at which the pruned route's energy advantage over small-from-scratch disappears.
The proposal fails if any of three results holds. The predictor might miss the sweet band in most noisy settings. Small-from-scratch might dominate at matched energy. Or the backfire elasticity for every pruning route might sit well below 1, meaning that even modest demand response erases the savings. The last outcome would still be a useful negative result for how compression is reported.
References
- Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks
- Sparse Double Descent: Where Network Pruning Aggravates Overfitting
- Holistically Evaluating the Environmental Impact of Creating Language Models
- Energy Considerations of Large Language Model Inference and Efficiency Optimizations
- Deep Double Descent: Where Bigger Models and More Data Hurt
- BurstGPT: A Real-World Workload Dataset to Optimize LLM Serving Systems
- DSD$^2$: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?
- Memorize to generalize: on the necessity of interpolation in high dimensional linear regression