Automated pruning shrinks a trained neural network by choosing how many channels to remove from each layer. Bayesian optimization (BO) is a natural tool here: it fits a statistical model to a few expensive trials and uses that model to pick the next trial. Its limits are well known. With one pruning ratio per layer the search space has tens to hundreds of dimensions, and standard BO models stop learning in that regime. The best BO pruning method, BOCR, handles this by grouping similar layers and searching one ratio per group. This note argues that part of the difficulty is self-inflicted. The objective the optimizer sees, "accuracy of the masked network," is distorted by the network's normalization layers. Batch normalization adds a learned shift to every channel, so a channel that has been "pruned" by zeroing its input still outputs a nonzero constant. Its running statistics also go stale as soon as upstream channels disappear. The optimizer is therefore scoring a network that differs from the one that will be deployed, and the scores contain configuration-dependent noise that a surrogate cannot fit.
The proposal is to make each pruning trial faithful to deployment and then check whether high-dimensional BO, run directly on per-layer or per-channel-group ratios, still needs dimensionality reduction. A faithful trial has three parts. Masked channels emit exact zeros after normalization. Any constant leakage that the deployed network would keep is folded into downstream biases. Statistics are recomputed for the specific configuration. If this framing is right, a properly initialized standard Gaussian process searching the full per-layer space should match or beat clustered search at equal budget, and its reported savings should survive physical removal of the channels.
Core Claim
Work on sparsity-preserving normalization has shown how normalization undoes zeros. Group Pruning Using a Bounded-ℓp Norm points out that when a gate zeroes a channel's input, batch normalization maps it to
$$y = \beta - \gamma \mu / \sigma,$$
where $\mu,\sigma$ are the stored channel mean and standard deviation and $\gamma,\beta$ the learned scale and shift. A zeroed channel therefore still feeds a constant into the next layer. Variational CNN Pruning observes the same leakage when scaling factors approach zero. PAT shows that layer normalization does the equivalent in pruned language models.
Separately, the heat-layout surrogate benchmark of Chen et al. found that batch statistics become unstable when the fraction of nonzero entries varies from batch to batch. That is the "fluctuating density" problem, and it describes exactly what happens when each BO trial removes a different set of channels.
These findings motivate the proposal, but none of them have been examined from the optimizer's side. BOCR treats the masked network as an unexamined black box. Our claim has two parts, and each can fail on its own. First, under a naive masked-evaluation protocol, the gap between masked accuracy and physically-pruned accuracy is large and varies systematically with the configuration. Second, removing that gap makes the objective smoother and closer to axis-aligned, so that sparse-prior and well-initialized GPs can model it without layer clustering.
Research Program
We will build a deployment-faithful evaluation wrapper for pruning trials and treat the choice of evaluation protocol as an experimental variable. There are four protocols.
The naive protocol masks channels and evaluates with the stored statistics. We believe this is the regime BOCR-style searches operate in, though the BOCR paper does not specify its evaluation details. The gated protocol multiplies the normalized output by the channel mask, so pruned channels are exactly zero.
The folded protocol is used when the deployed network would legitimately keep the constant. It absorbs the post-activation constant into the next convolution's bias. Folding is exact only away from zero-padded borders, so we report the residual border error separately.
The full faithful protocol adds per-configuration recalibration of normalization statistics on a fixed calibration set of a few hundred training images. Every protocol is also checked against ground truth: the network rebuilt with the channels physically removed, timed for real latency and counted for real FLOPs.
On the optimizer side, the program goes up a ladder of search dimensionality. The first rung is BOCR's clustered space. The second is one ratio per prunable layer, which is tens of dimensions for ResNet56 and MobileNetV2. The third is one ratio per channel group within each layer, which reaches hundreds of dimensions. That third rung is the regime where SAASBO and the scaled-likelihood-plus-subspace-perturbation recipe of Understanding High-Dimensional Bayesian Optimization (MSR) were designed to work.
SAASBO places a sparsity prior on per-dimension relevance and samples it with MCMC. MSR initializes GP length scales at $0.1\sqrt{d}$ and perturbs about 20 coordinates around the incumbent. We reuse both unchanged, because the point is to change the objective, not the optimizer.
Our diagnostics follow SAASBO's own diagnostic: held-out predicted-versus-true correlation of the surrogate, which SAASBO used to show that dense GPs give "flat" predictions on MOPTA08. We will also measure evaluation noise as the spread across repeated evaluations of one configuration with different calibration subsets. Finally, we will measure fitted relevance sparsity, meaning how many dimensions SAASBO keeps active under each protocol. Our hypothesis predicts fewer active dimensions and higher held-out correlation under the faithful protocol.
Baselines
The naive reference point has two parts: uniform pruning, with one ratio applied to every layer at the target FLOPs, and random search over per-layer ratios at the same evaluation budget. Any method that fails to beat random search on the per-layer space is not exploiting structure.
The strongest existing method is BOCR. It reports 92.53% top-1 on CIFAR-10 ResNet56, a 2.2% gain over naive BO, with a search budget of about 800 seconds. It also reports 49.34% top-1 for ImageNet MobileNetV1 at 50% FLOPs and 52.86% for MobileNetV2 at 60% FLOPs. These ImageNet figures are low for those architectures, which suggests they are measured before full fine-tuning; that is exactly where stale statistics should matter most. We will run BOCR with its clustering and rollback under both the naive and the faithful protocol, using the same number of trials, the same calibration and validation split, and the same post-search fine-tuning schedule applied to every method's final configuration. Beating BOCR means one of two things. Either a non-clustered per-layer search under the faithful protocol matches or exceeds BOCR's final fine-tuned accuracy at equal FLOPs with no more trials, or it matches BOCR on accuracy at measurably lower physically-measured latency.
Naive per-layer BO with a default-initialized GP is the configuration BOCR improved upon, so it anchors the claim that dimensionality was the bottleneck. MSR and SAASBO on the per-layer and per-group spaces are the strongest high-dimensional optimizers in the file, and we run both under every protocol. We also include CMA-ES and TuRBO, which LassoBench found often beat embedding-based BO; on its Leukemia task CMA-ES reached a best MSE of 0.015.
The key ablation crosses protocol with optimizer. Gating alone, recalibration alone, and both together isolate which normalization repair carries the effect.
Why not just…?
Why not just recalibrate batch-norm statistics after masking? This is the most serious threat. Recalibration is cheap, and it may account for most of the noise reduction. It does not stop gated-out channels from leaking a constant, so it cannot close the gap with the physically-pruned network by itself. Even so, it is possible that recalibration alone delivers nearly all of the gain in optimizer performance. The gating-versus-recalibration ablation settles this. If recalibration alone suffices, the paper's conclusion narrows to "the evaluation protocol, not intrinsic dimensionality, limits pruning BO," and that is still the claim being tested.
Why not just keep BOCR's layer clustering? Clustering forces structurally similar layers to share a ratio. It imposes that constraint because the optimizer could not model the full space, not because the constraint is known to be good. If faithful evaluation makes per-layer GPs predictive, the constraint becomes optional. The surrogate-correlation diagnostic and the head-to-head at equal budget show whether that happens. If BOCR still wins under the faithful protocol, the intrinsic-dimensionality explanation stands, and we will say so.
Why not just use CMA-ES or TuRBO and skip surrogate modeling? LassoBench shows these can outperform elaborate BO in high dimensions. However, pruning trials with recalibration and validation are expensive, and evolutionary methods typically need many more of them. We include both methods at the same trial budget. If they win under the faithful protocol, that still supports the main claim, because the protocol fix should help any optimizer that is misled by a distorted objective.
Why not swap batch norm for group or layer normalization? Pretrained pruning targets use batch norm, and retraining with a different normalization changes the model being pruned. Group and layer normalization also subtract a mean and add a shift, so they still turn zeros into nonzero values, as PAT shows for layer normalization.
Proposed Experiments
The experiments use BOCR's three settings: ResNet56 on CIFAR-10, and MobileNetV1 and MobileNetV2 on ImageNet. We search at BOCR's FLOPs targets on all three rungs of the dimensionality ladder, with five seeds per cell.
The first decisive check measures the deployment gap. We sample a few hundred random configurations and compute naive masked accuracy minus physically-pruned accuracy for each. If this gap is small and roughly constant, the core claim fails and the project stops there.
The second check compares surrogate quality across protocols. We fit SAASBO and MSR GPs to the same configurations under each protocol and compare held-out rank correlation and the number of active dimensions.
The third check is optimization performance: best physically-pruned accuracy at equal FLOPs, plotted against trial count, after a shared fine-tuning schedule. This is also where the comparison against BOCR's reported figures is made.
The fourth check tests whether faithful evaluation changes which configurations get chosen. We ask whether the configurations selected under the naive protocol disproportionately prune layers whose batch-norm shifts leak the largest constants. That would be the direct signature of the optimizer exploiting leakage.
References
- Bayesian Optimization with Clustering and Rollback for CNN Auto Pruning
- Group Pruning Using a Bounded-ℓp Norm for Group Gating and Regularization
- Variational Convolutional Neural Network Pruning
- PAT: Pruning-Aware Tuning for Large Language Models
- A Deep Neural Network Surrogate Modeling Benchmark for Temperature Field Prediction of Heat Source Layout
- High-dimensional Bayesian optimization with sparse axis-aligned subspaces
- Understanding High-Dimensional Bayesian Optimization
- LassoBench: A High-Dimensional Hyperparameter Optimization Benchmark Suite for Lasso