Single-Cycle Compression of Deep Neural Networks

This direction is merged into another direction: one-shot pruning. Its last version stays readable and citable here.

Version 1 of 122 papers30 conceptsfrozen 2026-10-04

The traditional recipe for making a neural network smaller is a tedious, multi-stage cycle: train a dense model to full accuracy, prune away the least important weights, and then fine-tune the remaining parameters to recover the lost performance. If the target size isn't met, the process repeats. This "train-prune-retrain" loop is computationally expensive and often results in compounding errors as each layer is optimized in isolation.

A growing body of research is moving toward one-shot or single-cycle compression. The core intuition is that the optimal sub-network architecture and its weights should be discovered simultaneously, rather than sequentially. By treating pruning as a global optimization problem or an integral part of the initial training phase, researchers are producing deployment-ready models in a single pass.

Moving Beyond Layer-by-Layer Pruning

Traditional pruning often treats layers as independent silos. However, layers in a deep network are deeply codependent; removing a filter in layer $N$ changes the input distribution for layer $N+1$.

To address this, Eunhyeok Park and Yong-Deok Kim proposed a one-shot pipeline that applies Variational Bayesian Matrix Factorization (VBMF) to the entire architecture at once. By selecting ranks and decomposing tensors across all layers simultaneously, their framework avoids the compounding errors that plague sequential methods. Similarly, Xavier Suau and colleagues developed Principal Filter Analysis (PFA), which identifies redundancy by looking at the correlation of activations across the whole network. This "global" view allows for massive compression (up to 8x for VGG-16) in a single pass because it accounts for how filters behave in concert rather than just their individual weight magnitudes.

Pruning as a Trajectory in the Loss Landscape

If we can identify which directions in the weight space don't affect the final loss, we can "prune" the model during training without ever losing accuracy. This is the intuition behind Structured Directional Pruning (SDP).

Yinchuan Li and Xiaofeng Liu demonstrated that neural networks have "flat valleys" in their loss landscapes—regions where parameters can be changed significantly with negligible impact on performance. By constraining parameter reductions to these constant-loss planes during the initial training phase, their AltSDP solver produces a compressed model in a single training run. This eliminates the need for any post-training fine-tuning because the model never "broke" during the pruning process.

Bridging the Discretization Gap

One of the hardest parts of one-shot pruning is the "discretization gap." Pruning is a binary decision (keep or drop), but neural network training relies on smooth, differentiable gradients.

Weihao Lin and Shengji Tang addressed this with S2HPruner, a "soft-to-hard" distillation paradigm. Instead of a sudden cut, they train a "soft" proxy network alongside a "hard" pruned version. By using bidirectional distillation, the soft network teaches the hard network how to maintain performance despite its missing pieces. This allows the architecture discovery and parameter training to happen in a single, unified stage.

A similar "online" approach was explored by Shaohui Lin through Online Ensemble Distillation (OED). By creating a strong "teacher" on-the-fly from multiple student branches, the network learns to prune entire blocks in one cycle, bypassing the iterative "prune-then-fine-tune" iterations entirely.

One-Shot Scaling for Large Language Models

The shift toward one-shot methods is even more critical for Large Language Models (LLMs), where the cost of a single training run can be millions of dollars. Iterative pruning is not just slow for LLMs; it can be detrimental.

Yao Lu and Zeyu Wang found that one-shot structural pruning is actually superior to iterative methods for LLMs because it avoids "catastrophic forgetting"—a phenomenon where step-by-step removal of layers causes the model to lose fundamental reasoning capabilities that are hard to recover. Their research suggests that simply cutting the last 25% of layers and performing a single recovery fine-tuning is more effective than complex iterative schemes.

This is echoed by Bo-Kyeong Kim, who showed that "depth pruning" (removing entire transformer blocks) in a single step provides linear speedups on modern GPUs, whereas traditional "width pruning" often results in irregular matrix shapes that offer no real-world performance gains.

The Efficiency of Early Discovery

The most recent frontier in this area is starting from scratch. Deepak Ghimire and Jaesik Park introduced OCSPruner, which performs structured pruning in a single training cycle starting from random initialization.

The intuition here is "stability-driven search": the optimal sub-network architecture can be identified very early in the training process. Once the architecture is locked in, the remaining computational budget is spent optimizing only the necessary parameters. This turns model compression from a post-processing chore into an efficient, front-loaded discovery process.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_single_cycle_compression_of_deep_neural__2e5a38b1,
  title = {Single-Cycle Compression of Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-cycle-compression-of-deep-neural-networks/rdr_2e5a38b18955460481555eaab2e57413},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_single_cycle_compression_of_deep_neural__2e5a38b1_v1,
  title = {Single-Cycle Compression of Deep Neural Networks},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/single-cycle-compression-of-deep-neural-networks/rdr_2e5a38b18955460481555eaab2e57413/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}