EvoP: Robust LLM Inference via Evolutionary Pruning

Shangyu Wu, Hongchao Du, Ying Xiong, Shuai Chen, Tei-Wei Kuo, Nan Guan, Chun Jason Xue

Abstract

As Large Language Models (LLMs) scale to hundreds of billions of parameters, their deployment on resource-constrained hardware is hindered by significant memory and latency overheads. Model pruning offers a solution by removing redundant components, yet existing layer-wise pruning methods often suffer from sub-optimal search heuristics and biased calibration data. We present EvoP (Evolutionary Pruning), a framework that utilizes an evolutionary algorithm to navigate the complex search space of layer-wise pruning and a cluster-based sampling strategy to ensure semantic diversity in calibration. Our results demonstrate that EvoP achieves superior accuracy and hardware-friendly speedups compared to current state-of-the-art methods.

Introduction

The deployment of LLMs on edge devices and standard servers is increasingly difficult due to the sheer size of modern architectures. While pruning techniques aim to alleviate these costs, "layer-wise" pruning—the removal of entire blocks—is particularly attractive because it provides direct hardware speedups without requiring specialized kernels. However, current layer-wise approaches typically rely on greedy heuristics that fail to find global optima in large search spaces. Furthermore, the small calibration datasets used to guide these methods are often sampled randomly, leading to patterns that may not generalize across diverse linguistic tasks.

EvoP addresses these challenges through two primary innovations: a robust evolutionary search strategy and a semantically aware data sampling technique.

The EvoP Framework

1. Cluster-based Calibration Dataset Sampling (CCDS)

Pruning requires a calibration dataset to evaluate which layers are least critical to model performance. Standard methods often sample this data randomly, which can introduce bias toward specific topics or styles. EvoP introduces CCDS to mitigate this. The process involves:

  • Chunking a raw dataset into smaller segments.
  • Generating semantic embeddings for these segments using a pre-trained BERT model.
  • Clustering the embeddings via KMeans.
  • Sampling equally from each cluster to form the final calibration set.

This ensures that the calibration data represents a broad spectrum of linguistic patterns, making the resulting pruning pattern more robust and generalizable.

2. Evolutionary Pruning Pattern Searching (EPPS)

To move beyond the limitations of greedy search, EvoP treats the pruning configuration as a "genome"—a binary vector where each bit represents whether a specific layer is kept or removed. The EPPS component employs an evolutionary algorithm to optimize this genome:

  • Selection, Crossover, and Mutation: The algorithm iteratively evolves a population of pruning patterns to find the one that minimizes perplexity on the calibration set.
  • Double Mutation Strategy: To maintain a strict target sparsity (e.g., removing exactly 20% of layers), EvoP uses a specialized mutation operator that flips bits in pairs, ensuring the total number of pruned layers remains constant throughout the search process.

Experimental Evaluation

Setup

The authors evaluated EvoP across several prominent LLM architectures, including Llama-2-13b, OPT-13b, and OPT-30b. Performance was measured using:

  • Perplexity: Evaluated on WikiText-2 and C4 datasets.
  • Downstream Accuracy: A suite of five benchmarks from the LM-Harness library, including ARC-Easy, ARC-Challenge, HellaSwag, PIQA, and Winogrande.

Key Results

EvoP consistently outperformed existing state-of-the-art (SOTA) methods across various metrics:

  • Accuracy Gains: On Llama-2-13b at 20% sparsity, EvoP achieved an average downstream score of 0.62, surpassing the previous SOTA layer-pruning method, SLEB (0.61), and significantly outperforming channel-wise methods like SliceGPT (0.45).
  • Generalization: By utilizing CCDS, EvoP demonstrated lower perplexity on out-of-domain datasets, proving that diverse calibration data prevents the model from overfitting to specific text types.
  • Hardware Efficiency: Because EvoP removes entire layers, it achieves linear speedups on standard GPUs. For instance, it delivered a 1.26x inference speedup on OPT-30b at 20% sparsity while maintaining higher accuracy than its competitors.

Discussion and Insights

The "Greedy" Trap

The research highlights a widening gap between greedy heuristics and the global optimum as model sparsity increases. While greedy methods are computationally cheap, they are prone to local optima. Evolutionary search, though more intensive, effectively navigates the combinatorial complexity of large-scale models.

Semantic Diversity

A critical finding of this work is that different semantic clusters "prefer" different pruning patterns. By forcing diversity in the calibration set, EvoP identifies layers that are truly redundant across the entire language distribution rather than just a subset of it.

Limitations and Future Work

While EvoP provides significant improvements, it does introduce a higher "one-time" computational cost during the initial search phase compared to greedy methods. Future research could explore combining evolutionary search with finer-grained pruning (such as individual heads or neurons) to achieve even higher compression ratios. Additionally, testing the framework on "frontier" models exceeding 70B parameters will be essential to validate its scalability.

Conclusion

EvoP represents a significant step forward in making LLMs more accessible. By combining semantically diverse data sampling with a powerful evolutionary search strategy, it provides a robust framework for layer-wise pruning that balances high accuracy with tangible hardware performance gains.

Summary updated

BibTeX
@inproceedings{wu2025evop,
  author = {Shangyu Wu and Hongchao Du and Ying Xiong and Shuai Chen and Tei-Wei Kuo and Nan Guan and Chun Jason Xue},
  title = {{EvoP: Robust LLM Inference via Evolutionary Pruning}},
  booktitle = {NLPCC},
  year = {2025},
  doi = {10.1007/978-981-95-3343-5\_36},
}
Figure