One-Shot Pruning for Large Scale Models

This direction is merged into another direction: one-shot pruning. Its last version stays readable and citable here.

Version 1 of 114 papers17 conceptsfrozen 2026-10-04

As neural networks scale to hundreds of billions of parameters, the traditional "train-prune-retrain" cycle has become a computational bottleneck. Retraining a Large Language Model (LLM) or a massive recommendation system after pruning is often as expensive as the initial training.

Recent research has shifted toward one-shot (or single-shot) pruning. The core intuition is that we can identify and remove redundant parameters in a single pass using a tiny "calibration" dataset, bypassing the need for expensive fine-tuning while preserving the model's emergent reasoning and performance.

The Intuition: Finding the "Fair Contribution"

In a dense network, not every parameter contributes equally to the final output. Traditional pruning often relies on simple metrics like weight magnitude (removing the smallest weights). However, in complex models like Transformers or Mixture-of-Experts (MoE), a small weight might still be critical for a specific reasoning path.

To solve this, researchers like Hung Vinh Tran and Hung Tran propose the Shaver framework, which views pruning through the lens of cooperative game theory. By calculating Shapley values, Shaver determines the "fair contribution" of each embedding. This creates a universal ranking of parameter importance, allowing a model to be instantly "shaved" to fit any memory budget—from a high-end server to a mobile device—without ever needing to be retrained.

Measuring Long-Range Impact

One-shot pruning works because it moves beyond local importance to global sensitivity. Instead of looking at a weight in isolation, frameworks like OPTIN, developed by Samir Khaki and Konstantinos N. Plataniotis, measure a parameter's "trajectory."

This approach analyzes how the removal of a weight ripples through downstream layers. By approximating the optimal sub-network directly from pre-trained weights, OPTIN achieves state-of-the-art efficiency across both NLP and Computer Vision domains in a single pass. Similarly, Yunwei Li and colleagues introduced E-Sparse, which uses information entropy to identify critical data channels. By shuffling these channels to fit hardware-friendly N:M sparsity patterns, they achieve significant inference speedups without the accuracy drop typically seen in training-free methods.

Pruning by Listening to the Router

The rise of Mixture-of-Experts (MoE) models—where only a fraction of the network is active for any given input—presents a unique opportunity for one-shot pruning. Since MoE models already use a "router" to select experts, we can use that router's own logic to find redundancies.

  • MoE-Pruner: Yanyue Xie and Zhi Zhang developed a method that uses "hints" from the model's router. By observing which experts the router ignores during a single forward pass of calibration data, they can prune models like Mixtral-8x7B to 50% sparsity while retaining 99% of performance.
  • STUN: For even larger models, Jaeseong Lee 0002 and Seung-won Hwang proposed Structured-Then-Unstructured (STUN) pruning. This two-stage approach first removes entire redundant experts (structured) and then prunes individual weights (unstructured). This allowed them to compress the 480B-parameter Snowflake Arctic model in just two hours on a single GPU.

Adaptation Without Forgetting

A major challenge in one-shot pruning is maintaining performance as models adapt to new domains. Srikanth Malla and Chiho Choi introduced COPAL, which focuses on continual pruning.

COPAL uses sensitivity analysis on a minimal set of data samples to identify weights that are critical across multiple encountered datasets. This ensures that as the model is compressed for a new task, it doesn't "forget" the knowledge it previously held. As noted in the survey by Wenxiao Wang and Yicong Luo, these tuning-free methods are essential for preserving the "emergent reasoning" abilities of LLMs, which are often fragile and easily destroyed by aggressive, uninformed pruning.

Why It Works: The Calibration Pass

The "secret sauce" of all these methods is the calibration dataset. Instead of the millions of samples required for training, one-shot pruning typically uses a tiny, representative set of data (often just a few hundred samples).

During a single forward pass, the algorithm collects statistics—activations, gradients, or routing frequencies. Because the model is already highly optimized, these statistics are sufficient to identify the "dead wood." By focusing on the most robust importance metrics—whether Shapley values, entropy, or router hints—researchers can approximate the optimal sub-network without ever needing to run a backpropagation step.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_one_shot_pruning_for_large_scale_models_72a5575a,
  title = {One-Shot Pruning for Large Scale Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-shot-pruning-for-large-scale-models/rdr_72a5575a2c55406fa0c3c0e77ad79911},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_one_shot_pruning_for_large_scale_models_72a5575a_v1,
  title = {One-Shot Pruning for Large Scale Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/one-shot-pruning-for-large-scale-models/rdr_72a5575a2c55406fa0c3c0e77ad79911/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}