Evolutionary biologists separate two kinds of DNA change in a gene. Synonymous changes alter the DNA letters but not the protein. Non-synonymous changes alter the protein. Selection mostly ignores the first kind, so synonymous changes pile up at a steady rate and act as a clock; the rate of synonymous substitutions per synonymous site is written $K_s$. Selection filters the second kind. The ratio of the two rates is the classic test for whether a gene is under functional constraint: a low ratio means selection is removing most function-altering changes. We carry this ratio over to neural network training. Over a short window of training, we split each layer's weight movement into a part that changes the network's outputs and a part that does not, and normalize each part by how many directions are available to it. The resulting per-layer ratio then controls a progressive pruning schedule. A pruning schedule decides how fast, and in which layers, weights are removed during training. Layers whose weights wander freely in output-neutral directions are carrying spare capacity and can be pruned faster. Layers whose movement concentrates on output-changing directions are under "selection" and should be left alone for now.
This matters because the schedules in wide use are blind to what the network is doing. The cubic schedule of To Prune, or Not to Prune sets sparsity purely from the training step index. Yet Sparse Double Descent shows that under label noise, moderate gradual pruning drives the network through a "critical phase" in which it overfits more rather than less. Functional Degeneracy in Neural Networks shows that saturated networks have a large neutral subspace that magnitude pruning fails to exploit. A signal that measures constraint directly, layer by layer and over time, could decide both when to prune and where. That is the gap a step-count clock cannot fill.
Core Claim
A windowed, per-layer selection ratio, computed from a low-rank estimate of the output-sensitive subspace, is a better control signal for progressive pruning than elapsed steps, loss curves, or static redundancy scores. Using it to set layer budgets and to pause or speed up the global schedule should do three things. It should match or beat fixed schedules on clean data at high sparsity. It should shrink or remove the critical-phase dip on noisy-label data. It should also retire most of the hand-set tempo hyperparameters that growing-regularization methods need.
Concretely, let $\Delta\theta_\ell$ be the change in layer $\ell$'s weights over a window of $W$ steps. Let $P_\ell$ be the projector onto the top-$k_\ell$ eigendirections of that layer's output-sensitivity matrix. This matrix is the behavioral Hessian of Matveev et al., which we approximate with the low-rank empirical Fisher $H=\tfrac{1}{n}A^\top A$ used by CHITA. Let $d_\ell$ be the layer's parameter count. The selection ratio is
$$\omega_\ell=\frac{\lVert P_\ell\,\Delta\theta_\ell\rVert^2/k_\ell}{\lVert (I-P_\ell)\,\Delta\theta_\ell\rVert^2/(d_\ell-k_\ell)},$$
which is movement per output-changing direction divided by movement per neutral direction. Dividing by $k_\ell$ and $d_\ell-k_\ell$ plays the same role as counting "per site" in $K_s$. Without it, large layers would look neutral simply because most of their directions are.
We choose $k_\ell$ adaptively, as the recovery rank at a fixed tolerance, following Matveev et al. The biological analogy also warns of two failure modes. One is saturation: a clock stops being informative once changes overwrite each other. The other is confounding: Chen et al. show that local tandem duplication can mimic genome-wide doubling unless context is checked. We therefore require $\omega_\ell$ to be stable across window lengths before trusting it. We also report where it fails.
Research Program
The first stage is purely measurement. We train dense and gradually pruned ResNet56 models on clean and noisy-label CIFAR-10 and CIFAR-100 and track $\omega_\ell$ for every layer over training. The central descriptive question is whether $\omega$ rises, across layers or in specific layers, at the onset of the critical phase. Sparse Double Descent reports that the distance traveled from initialization spikes in that phase. Our hypothesis is that this extra travel is non-synonymous, meaning the network is spending real capacity on memorizing noisy labels. If $\omega$ does not move there, the scheduling idea loses its main motivation, and we will report that result.
The second stage builds the scheduler. Following Zhu & Gupta, the cubic curve serves only as an upper envelope on global sparsity. Within it, the current step's pruning budget goes to layers in proportion to $\exp(-\beta\,\omega_\ell)$. The global increment is paused whenever the median $\omega$ exceeds its running baseline by a set margin. Weight scores within a layer remain ordinary magnitude or Fisher scores. The new ingredient is the tempo and the allocation, not the choice of individual weights.
SGD noise can inflate the neutral term. We therefore smooth $\Delta\theta$ with the Polyak averaging that SEVEN applies to gradient moments for Transformer pruning. We refresh the low-rank Fisher every $W$ steps using a few hundred examples, which keeps overhead to a small fraction of training compute; we will measure that overhead.
The third stage transfers the idea to large language models, where training is too expensive for the full scheduler. There we use $\omega_\ell$ only as a layer-allocation signal, computed during a short calibration fine-tune. It competes directly with the layerwise redundancy rule of Maximum Redundancy Pruning.
A side analysis borrows the most familiar tool of $K_s$ work, the histogram whose peaks mark genome-wide duplication events (Yue et al.; Ma et al.). For pairs of near-duplicate neurons, we plot the distribution of the neutral-subspace distance between them. We then ask whether redundancy forms in discrete bursts early in training, which would show up as sharp peaks, or accumulates continuously. We treat this as a diagnostic that could motivate later work, not as a claimed contribution.
Several related lines of work differ from ours in specific ways. Synonymous Variational Inference uses "synonymous" to mean perceptually equivalent images and slices latent channels. Injecting Knowledge via Synonymous Substitution uses synonymous mutations as data augmentation. The brain-inspired sparse training work (Zhang et al.) prunes and regrows weights according to network topology. None of these measures constraint on weight movement as a rate. Null-LoRA projects updates into a null space for fine-tuning. We use the split in the opposite direction: we measure it rather than enforce it.
Baselines
The naive reference points are the dense model trained with an identical budget and one-shot magnitude pruning to the target sparsity followed by fine-tuning. Together they bound the gain that any schedule can claim.
The main schedule baseline is the cubic gradual schedule of Zhu & Gupta. It is the default everywhere and is our envelope, so beating it isolates the value of adaptive tempo and allocation. We run it with the same start and end steps, the same pruning frequency, and a uniform layer budget. Separately, we also run it with the best global versus per-layer setting found in an equal-size tuning sweep.
The strongest progressive competitor for structured pruning is Growing Regularization. GReg-1 reports 93.06% CIFAR-10 accuracy for ResNet56 at 2x speedup and 85.97% at 19x speedup, against 82.79% for one-shot pruning. It is the strongest competitor because it already removes pruning shock by gradually increasing a penalty. We reproduce it with the paper's hyperparameters, then give it the same number of tuning trials we give ourselves over its ceiling $\tau$, step $\delta\lambda$ and interval $K_u$. Beating it means higher accuracy at 19x speedup averaged over three seeds with non-overlapping confidence intervals, or equal accuracy with no tempo hyperparameters tuned. We also include ASRFP, which decays filter scales toward zero and reports a 0.64% drop for ResNet-110 at 40% pruning against 1.42% for soft filter pruning.
For weight selection, CHITA++ reports 47.45% ImageNet accuracy for MobileNetV1 at 80% sparsity. We run it in its multi-stage mode, and we also run a hybrid that uses CHITA's selection inside our schedule. This separates the effect of the scheduler from the effect of the weight-scoring rule.
For language models, the bar is MRP. On LLaMA2-7B at 70% Wanda sparsity it reports WikiText2 perplexity of 23.54, against 30.47 for OWL and 80.40 for uniform allocation. We must match its calibration data size to claim a win. For the noisy-label regime, we found no reported result on Lacuna for gradual-pruning accuracy at specific noise rates in the sparse double descent paper, so we reproduce its setting ourselves.
The key ablation replaces $P_\ell$ with a random subspace of the same rank $k_\ell$. If random-subspace ratios schedule as well as Fisher-derived ones, the effect comes from raw update norms, not from the synonymous/non-synonymous split.
Why not just…?
Why not just use the cubic or exponential schedule, which cost nothing? On clean data they may well be enough. Zhu & Gupta lose no accuracy at 50% sparsity on InceptionV3, and FORCE (Progressive Skeletonization) keeps about 90% CIFAR-10 accuracy at 99% sparsity. We expect our advantage to be concentrated at extreme sparsity and under label noise, where step-based schedules walk into the critical phase. If the clean-data gain is nil, we will say that the method is a noisy-regime tool.
Why not just adapt the tempo to validation loss or gradient norm? This is the most serious threat. It is cheap and also reacts to training dynamics. Loss, however, is a single global signal and cannot allocate budget across layers. Under label noise, training loss keeps falling right through the critical phase. We include a loss-plateau scheduler and a per-layer gradient-norm scheduler as baselines. If either matches ours on noisy CIFAR, the subspace split is unnecessary and the paper becomes a negative result about it.
Why not just merge near-duplicate neurons by activation similarity? Similarity finds redundant pairs at one moment but says nothing about whether a layer is currently under constraint, and constraint is what should set pruning speed. That said, similarity-based allocation could capture most of the benefit. We therefore run it as an allocation baseline under the same cubic envelope.
Why not just use CHITA or Fisher-based scores and skip the scheduling question? CHITA answers a different question: which weights to remove at a given moment. It does not decide when. The CHITA-inside-our-schedule hybrid tests directly whether the two combine or whether one makes the other redundant.
Proposed Experiments
The decisive experiment is noisy-label CIFAR-10 and CIFAR-100 with ResNet18 and ResNet56 at 20% and 40% symmetric label noise, sweeping target sparsity from 50% to 99%. The main outcome is the depth and width of the critical-phase dip in test accuracy under each scheduler. The prediction is that $\omega$-scheduling flattens the W-shaped curve, while the random-subspace ablation and the loss-based scheduler do not.
On clean data, we compare structured ResNet56 on CIFAR-10 at 2x and 19x speedup against GReg and ASRFP. We compare unstructured MobileNetV1 on ImageNet at 75% and 80% sparsity against the cubic schedule and CHITA++. For LLaMA2-7B at 70% sparsity, we compare against MRP and OWL on WikiText2 perplexity.
Across all settings, we report three seeds, wall-clock overhead, and sensitivity to window length $W$ and rank tolerance. We also check whether $\omega$ remains informative at very high sparsity, where the analogy to clock saturation predicts it may break down.
References
- To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression
- Sparse Double Descent: Where Network Pruning Aggravates Overfitting
- Functional Degeneracy in Neural Networks: Measurement and Pruning
- Fast as CHITA: Neural Network Pruning with Combinatorial Optimization
- Chromosome-scale genome assembly of Codonopsis pilosula and comparative genomic analyses shed light on its genome evolution
- SEVEN: Pruning Transformer Model by Reserving Sentinels
- Maximum Redundancy Pruning: A Principle-Driven Layerwise Sparsity Allocation for LLMs
- Full-length transcriptome sequencing provides insights into the evolution of apocarotenoid biosynthesis in Crocus sativus
- Phylogenetic Analysis and Expression Patterns of Triterpenoid Saponin Biosynthesis Genes in 19 Araliaceae Plants
- Synonymous Variational Inference for Perceptual Image Compression
- Injecting Knowledge into Biomedical Pre-trained Models via Polymorphism and Synonymous Substitution
- Brain network science modelling of sparse neural networks enables Transformers and LLMs to perform as fully connected
- Null-LoRA: Low-Rank Adaptation on Null Space
- Neural Pruning via Growing Regularization
- Softer Pruning, Incremental Regularization
- Progressive Skeletonization: Trimming more fat from a network at initialization