One-Shot and Single-Stage Model Pruning

228 concepts163 papers1 proposal

The traditional recipe for neural network compression follows a rigid, three-stage cycle: train a dense model to convergence, prune away the least important weights, and fine-tune the remaining network to recover lost accuracy. While effective for smaller architectures, this iterative paradigm breaks down when applied to modern large-scale models. As highlighted by Cheng et al. and Roux et al., running multiple cycles of pruning and retraining on models with billions of parameters is computationally and financially prohibitive.

To bypass this bottleneck, researchers have shifted toward one-shot and single-stage pruning. The core intuition is simple: instead of treating compression as a destructive post-processing step that requires extensive repair, we can either identify and compensate for redundant parameters in a single pass (one-shot post-training) or bake the pruning directly into the initial optimization process (single-stage integrated training).

The Mechanics of One-Shot Post-Training Pruning

How can we remove a massive percentage of a model's parameters in a single step without causing its accuracy to collapse? The secret lies in moving away from naive, localized heuristics (like magnitude-based pruning) toward global, interaction-aware metrics and fast local reconstruction.

Second-Order and Structural Insights

Simple magnitude pruning assumes weights close to zero are useless. However, a small weight might be crucial when interacting with other parameters. To capture these interactions without full retraining, Singh demonstrates that second-order Taylor approximations—specifically estimating the inverse Hessian matrix via the empirical Fisher information—can predict the global impact of removing specific weights in a single step.

Local Knowledge Reconstruction

When a layer is pruned, its output changes, causing an error that compounds across subsequent layers. Instead of running full backpropagation to fix this, one-shot methods use fast linear algebra to "reconstruct" the lost information layer-by-layer. For instance, Park et al. use knowledge-preserving linear adjustments to minimize the feature-map discrepancy caused by structured pruning. Similarly, Wang et al. propose a framework that globally identifies filter importance and reconstructs feature maps at key junction nodes in a single recovery phase, entirely bypassing layer-by-layer iterative retraining.

Proxy-Based and Geometric Selection

Rather than computing expensive gradients over large datasets, some one-shot approaches leverage structural properties or small calibration sets. Kwon et al. show that optimizing a small set of mask variables over a tiny fraction of data can compress transformers in minutes. Taking this further, Dong et al. utilize symbolic zero-cost proxies to rank and prune low-importance components without any post-pruning training, while Bhardwaj et al. use tropical geometry to remove redundant parameters while preserving the geometric shape of the decision boundaries.

Integrated Single-Stage Pruning: Sparsity During Training

An alternative to post-training pruning is to integrate compression directly into the training loop. This collapses the multi-stage pipeline into a single training session where the model naturally converges to a sparse state.

Sparsity-Inducing Regularization

By incorporating sparsity constraints directly into the optimization objective, models can learn which structures are redundant while they learn the task. Molchanov et al. achieve this by extending variational dropout to allow for unbounded dropout rates, forcing redundant parameters to zero during training so they can be cleanly thresholded out later. Lin et al. combine sparsity-inducing regularization with online ensemble distillation to remove entire structural blocks in a single training pass.

Co-Designing Weights and Masks

Instead of pruning at the very end, some methods prune dynamically as training progresses. Aketi et al. integrate structural pruning into the training phase by removing low-relevance channels at regular intervals, allowing the remaining parameters to adapt within the standard training budget. Hanson et al. show that integrating pruning directly into the training loop acts as a powerful regularizer, improving overall training efficiency by reducing the computational load of managed primitives in later stages.

To ensure that the model doesn't suffer a massive drop at the moment of pruning, Zhou et al. propose aligning weight values with their post-pruning optimal states during the initial optimization phase itself. This ensures that when the final cut is made, the model is already in a state that requires zero retraining.

Why It Works: The Intuition of Redundant Subspaces

Why can we skip the fine-tuning phase that was once thought essential? The answer lies in the geometry of overparameterized loss landscapes.

Deep neural networks contain massive internal redundancy; many neurons or filters track highly correlated features. When we prune a model iteratively, we are slowly carving a path through this landscape. One-shot and single-stage methods succeed because they treat the network's capacity globally rather than locally. By projecting activations into orthogonal subspaces or using mathematical closed-form solutions (like least-squares or singular value decomposition) to redistribute the responsibility of pruned weights onto the surviving weights, these methods shift the network to an equivalent, lower-dimensional representation instantly.

Whether by optimizing masks on a tiny calibration set or forcing the optimization objective to favor sparse topologies from day one, these techniques prove that the final fine-tuning step isn't a magical recovery phase—it was simply a slow, expensive way to do what clever linear algebra and structured regularization can achieve in a single pass.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_one_shot_pruning_791686dd,
  title = {one-shot pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-shot-pruning/rdr_791686dd35864fe8b27140eeea41472a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_one_shot_pruning_791686dd_v1,
  title = {one-shot pruning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-shot-pruning/rdr_791686dd35864fe8b27140eeea41472a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake