Abstract
Despite exceptional capabilities, Large Language Models (LLMs) still face deployment challenges due to their enormous size. Post-training structured pruning is a promising solution that prunes LLMs without the need for retraining, reducing computational overhead, and it is hardware-deployment friendly. However, the training-free nature of post-training structured pruning leads to significant performance degradation. We argue that the key to mitigating this issue lies in accurately determining the pruning rate for each layer. Meanwhile, we find that LLMs may have prior knowledge about their own redundancy. Based on this insight, we introduce Self-Pruner, an end-to-end automatic self-pruning framework for LLMs, which efficiently searches for layer-wise pruning rates. Specifically, Self-Pruner leverages LLMs to autonomously execute the entire evolutionary search process. In this process, LLMs are used to generate populations, select parent solutions, and perform crossover and mutation operations to produce offspring solutions. Extensive experiments demonstrate Self-Pruner's superior performance, pruning LLaMA-2-70B to 49B with only a 0.80% drop in accuracy and achieving a 1.39× speedup.
Introduction
As LLMs scale to hundreds of billions of parameters, deployment becomes increasingly difficult due to memory constraints and high inference costs. Structured pruning—removing entire components like attention heads or MLP neurons—offers a hardware-friendly path to efficiency. However, most current methods apply a uniform pruning rate across all layers, leading to a "performance cliff" where accuracy drops sharply.
Not all layers contribute equally to a model's reasoning capabilities; some are highly redundant, while others are critical. Manually determining the optimal pruning rate for each layer is a complex combinatorial problem. Self-Pruner addresses this by asking: Can we use the LLM's own internal knowledge of its architecture to automate this search?
Methodology: The Self-Pruner Framework
Self-Pruner treats the search for pruning rates as an evolutionary process, utilizing an LLM (such as GPT-4o) as the "evolutionary engine." The framework operates in four distinct stages:
- Population Initialization: The LLM is prompted to generate an initial set of diverse layer-wise pruning rate configurations based on its prior knowledge of model redundancy.
- Evaluation (Fitness): The target model is pruned using these configurations, and its "fitness" is measured by its perplexity on the WikiText-2 dataset.
- Evolutionary Operations via LLM: Instead of hard-coded mathematical formulas, the framework uses the LLM for crossover and mutation. The LLM receives the best-performing configurations and generates improved offspring through "chain-of-thought" reasoning.
- Iteration and Finalization: This cycle repeats for $N$ generations until the population converges on an optimal strategy, resulting in the final pruned model.
By using natural language prompts to guide the evolution, the framework eliminates the need for manually designed importance metrics or complex hyperparameter tuning.
Experimental Setup
The researchers evaluated Self-Pruner across a variety of models, including LLaMA-1 (7B, 13B), LLaMA-2 (7B, 13B, 70B), LLaMA-3 (8B), and Vicuna (13B). While WikiText-2 served as the fitness dataset during the search, the final models were evaluated across seven commonsense reasoning benchmarks: Winogrande, HellaSwag, BoolQ, ARC-Easy, ARC-Challenge, OpenBookQA, and PIQA.
Key Results and Insights
Self-Pruner demonstrates a superior trade-off between model size and performance compared to existing state-of-the-art methods like LLM-Pruner and Wanda-sp.
- High-Fidelity Compression: LLaMA-2-70B was compressed to 49B parameters (30% sparsity) with only a 0.80% drop in average accuracy, yielding a 1.39× speedup on an NVIDIA A100 GPU. Even at 50% sparsity (35B parameters), the model maintained respectable performance with only a 3.80% accuracy decrease and a 1.70× speedup.
- LLMs as Genetic Operators: Replacing traditional mutation functions with high-level natural language instructions makes the framework highly flexible. The results confirm that LLMs possess latent knowledge about their own redundancy, allowing the search to converge faster than random or purely mathematical approaches.
- The Importance of Non-Uniformity: The search consistently found that the best strategies are highly non-uniform; some layers can tolerate 80% pruning while others must remain nearly intact to preserve reasoning.
Conclusion and Future Work
Self-Pruner represents a significant step toward autonomous model optimization. While the evolutionary search is efficient, it still requires multiple inference passes for fitness evaluation. Future research may explore more granular self-compression—using LLMs to decide which specific components to prune rather than just layer-wise rates—and developing specialized fine-tuning methods to further "heal" the performance gap in pruned models.
Summary updated