Towards Efficient Automatic Self-Pruning of Large Language Models

Weizhong Huang, Yuxin Zhang, Xiawu Zheng, Fei Chao, Rongrong JiSubmission to ICLR 2025 · Rejected

TL;DR

Self-Pruner automates post-training structured pruning for Large Language Models (LLMs) by turning an LLM into the engine of an evolutionary search. Rather than using uniform pruning or hand-tuned heuristics across layers, Self-Pruner prompts an LLM (such as GPT-4o) to generate, mutate, and cross over layer-wise pruning configurations, enabling models like LLaMA-2-70B to be compressed by up to 50% with minimal accuracy degradation and up to 1.70× inference speedup.

What Problem Does This Paper Address?

Large Language Models have massive memory footprints and high inference latency. While post-training structured pruning is an attractive solution—it removes entire channels or heads to produce immediate hardware speedups without retraining—it usually leads to sharp accuracy drops.

A primary reason is the layer-wise pruning rate allocation:

  1. Standard approaches apply uniform pruning rates across all layers, ignoring the reality that different layers store vastly different amounts of redundant information.
  2. Existing non-uniform heuristics (such as outlier-based allocations like OWL) rely heavily on manual importance metrics and laborious hyperparameter tuning.
  3. Traditional evolutionary algorithms (EAs) can search for per-layer rates, but designing custom genetic operators (crossover, mutation) is difficult, and standard EAs take too many iterations to converge when evaluating LLMs is computationally heavy.

Methods

Self-Pruner reframes the search for optimal layer-wise pruning ratios as an evolutionary optimization executed entirely by an LLM via natural language prompting:

  • LLM-Driven Evolutionary Search:
  • Population Initialization: A prompt asks the LLM to propose an initial population of layer-wise pruning configurations summing to a target overall pruning ratio $\beta$, exploiting the LLM’s latent understanding of Transformer architecture redundancy.
  • Fitness Evaluation: Candidate configurations prune the model layers using Wanda-sp (to determine which specific neurons to remove inside each layer), and fitness is scored using model perplexity on the WikiText-2 validation set.
  • LLM Selection, Crossover, and Mutation: The current top candidates and their fitness scores are fed back into the LLM via prompt. The LLM acts as the genetic operator, autonomously selecting parents and generating new offspring configurations through high-level instructions rather than rigid mathematical operators.
  • Iteration: The process runs for a small number of generations ($N = 20$, population size $K = 30$) to efficiently converge on the best layer-wise pruning schedule.
Prompt LLM -> Initial Pruning Ratios -> Prune via Wanda-sp & Eval Perplexity
      ^                                                            |
      |_________ LLM Selection, Crossover & Mutation <_____________|

Datasets

  • Evaluation & Fitness:
  • WikiText-2: Used to evaluate language modeling perplexity, serving as the fitness objective during evolutionary search.
  • Zero-Shot Commonsense Reasoning: Evaluated using lm-eval-harness across 7 standard benchmarks: Winogrande, HellaSwag, BoolQ, ARC-Easy, ARC-Challenge, OpenBookQA (OBQA), and PIQA.
  • Fine-Tuning Analysis: A 10K-sample subset from Alpaca-GPT4 was used to test downstream parameter recovery via LoRA.
  • Note: No new datasets were introduced.

Key Insights and Ideas

  • LLMs as Genetic Operators: Instead of human experts designing domain-specific mutation rates and crossover boundaries, modern LLMs (e.g., GPT-4o) intuitively handle multi-dimensional constraint optimization in natural language, exploring trade-offs between early, middle, and late layers.
  • Better LLMs Yield Better Schedules: Ablations showed that stronger reasoners find better configurations: GPT-4o (60.79% accuracy on LLaMA-2-7B at 30% pruning) outperformed GPT-4 (59.87%) and GPT-3.5 (58.94%).
  • Prior Architectural Knowledge Accelerates Search: Using the LLM to generate the initial population significantly outperformed random initialization (16.25 vs. 17.43 perplexity), reducing the search rounds required to reach high-performing solutions.

Key Results

The authors evaluated Self-Pruner on LLaMA-1, LLaMA-2, LLaMA-3, LLaMA-3.1, and Vicuna models (7B to 70B parameters) against baselines like LLM-Pruner, Wanda-sp, and OWL:

  • LLaMA-2-70B Compression:
  • 30% Pruning (~49B parameters): Retained near-original performance, dropping only 0.80% in average zero-shot reasoning (from 73.81% to 73.01%) while achieving a 1.39× inference speedup on NVIDIA A100 GPUs. Perplexity rose by only 1.88 points.
  • 50% Pruning (~35B parameters): Incurred only a 3.80% drop in accuracy (70.01% vs. 73.81% dense) while delivering a 1.70× speedup and halving memory consumption from 128.5 GB to 64.6 GB. In contrast, Wanda-sp dropped to 60.63% accuracy.
  • Outperforming Baselines: On LLaMA-2-7B at 50% pruning, Self-Pruner achieved 51.35% average accuracy, outperforming LLM-Pruner (40.45%), Wanda-sp (36.76%), and OWL (39.06%).
  • Fine-Tuning Recovery: Applying a brief LoRA fine-tuning run on 50%-pruned LLaMA-2-7B boosted accuracy from 51.35% to 59.78% and halved perplexity (53.63 $\rightarrow$ 25.32).

Limitations and Future Work

  • Semi-Automated Compression: Self-Pruner searches for the layer-level pruning ratio, but still relies on an external heuristic metric (Wanda-sp) to decide which specific weights/neurons within a layer are removed. Fully end-to-end self-pruning remains an open challenge.
  • Severe Degradation at Extreme Ratios Without Tuning: While Self-Pruner significantly outperforms prior post-training techniques, high pruning ratios (e.g., 50%) without fine-tuning still degrade smaller models (e.g., 7B) considerably.
  • Evaluation Cost: Evaluating model perplexity on real hardware across tens of population members over multiple generations still incurs substantial GPU compute. Future work aims to incorporate efficient recovery fine-tuning and extend LLM-driven pruning to unstructured sparsity and quantization.

Summary updated

Abstract

Despite exceptional capabilities, Large Language Models (LLMs) still face deployment challenges due to their enormous size. Post-training structured pruning is a promising solution that prunes LLMs without the need for retraining, reducing computational overhead, and it is hardware-deployment friendly. However, the training-free nature of post-training structured pruning leads to significant performance degradation. We argue that the key to mitigating this issue lies in accurately determining the pruning rate for each layer. Meanwhile, we find that LLMs may have prior knowledge about their own redundancy. Based on this insight, we introduce $\textbf{Self-Pruner}$ an end-to-end automatic self-pruning framework for LLMs, which efficiently search layer-wise pruning rates. Specifically, $\textbf{Self-Pruner}$ leverages LLMs to autonomously execute the entire evolutionary search process to search for pruning rate configurations. In this process, LLMs are used to generate populations, select parent solutions from the current population, and perform crossover and mutation operations to produce offspring solutions. In this way, LLMs automatically generate and evaluate a large number of candidate solutions, effectively converging to find the pruning rate configurations with minimal human intervention. Extensive experiments demonstrate $\textbf{Self-Pruner}$'s better performance compared to existing state-of-the-art methods. Notably, $\textbf{Self-Pruner}$ prunes LLaMA-2-70B to 49B level with only 0.80% drop in accuracy across seven commonsense reasoning tasks, achieving a 1.39$\times$ speedup on NVIDIA A100 80GB GPU. Further pruning to 35B level resulted in only a 3.80% decrease in accuracy while obtaining a 1.70$\times$ speedup. Code is available in the supplementary material.
BibTeX
@misc{huang2025towards,
  author = {Weizhong Huang and Yuxin Zhang and Xiawu Zheng and Fei Chao and Rongrong Ji},
  title = {{Towards Efficient Automatic Self-Pruning of Large Language Models}},
  year = {2025},
  eprint = {2502.14413},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.14413},
}
Figure