QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models

Changhai Zhou, Yuhua Zhou, Yibin Wang, Shijie Han, Qian Qiao, Hongguang Li

Introduction

Large Language Models (LLMs) are notoriously resource-intensive, posing significant challenges for deployment on consumer-grade hardware. To address this, researchers often employ structured pruning—removing entire parameter groups like neurons or attention heads—and quantization—reducing numerical precision. However, combining these techniques is non-trivial. Structured pruning often leaves a model "unbalanced," where certain layers become disproportionately critical to performance. Applying uniform quantization across such a model can severely degrade accuracy. Furthermore, the fine-tuning required to recover performance is often memory-intensive, potentially negating the benefits of compression during the development phase.

The QPruner Framework

To overcome these hurdles, the authors introduce QPruner, a three-stage pipeline designed to optimize the trade-off between model size, memory footprint, and task performance.

The three-stage QPruner pipeline: Structured Pruning, Mixed-Precision Quantization with Bayesian Optimization, and Performance Recovery.
  1. Structured Pruning: The process begins by pruning the model using a dependency-graph approach. This identifies and removes less important parameter groups based on Hessian-based importance estimation.
  2. Mixed-Precision Quantization: Unlike standard uniform quantization, QPruner assigns varying bit-widths (e.g., 4-bit vs. 8-bit) to different layers based on their relative importance.
  3. Initial Allocation: Mutual Information (MI) is used to calculate the statistical dependence between a layer's output and the model's final prediction. Layers with high MI are prioritized for higher precision.
  4. Refinement via Bayesian Optimization: To reach an optimal configuration, a Gaussian Process-based Bayesian optimizer explores the discrete search space of bit-widths. This "probabilistic decision" process aims to maximize accuracy while adhering to a strict memory budget.
  5. Performance Recovery: The compressed model undergoes fine-tuning using Parameter-Efficient Fine-Tuning (PEFT) techniques, specifically LoftQ, which is optimized for initializing LoRA adapters for quantized weights.

Experimental Evaluation

The researchers evaluated QPruner using several open-source LLMs, including LLaMA-7B, LLaMA-13B, and Vicuna-7B. Performance was measured across seven standard commonsense reasoning benchmarks: BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA.

Results and Analysis

QPruner demonstrates a superior ability to maintain accuracy while significantly reducing memory requirements compared to standard pruning baselines.

Performance comparison of QPruner against LLM-Pruner across various pruning rates and benchmarks for LLaMA-7B and Vicuna-7B.

Key Findings:

  • Significant Memory Savings: QPruner consistently reduced memory usage by over 30% compared to standard half-precision (FP16) pruned models. For instance, at a 50% pruning rate, the memory footprint of LLaMA-7B dropped from approximately 23.9 GB to 16.6 GB.
  • Enhanced Accuracy: Despite the reduced precision, QPruner frequently outperformed full-precision pruned baselines. At a 50% pruning rate on LLaMA-7B, QPruner achieved an accuracy of 48.37% on BoolQ, significantly higher than the 43.76% achieved by the standard LLM-Pruner.
  • Importance-Aware Synergy: The results suggest that quantization, when applied via an importance-aware strategy, acts as a regularizer. This helps the fine-tuning process focus on the most vital parameters, leading to better performance recovery.
  • Optimization Robustness: Ablation studies confirmed that the Bayesian Optimization step (QPruner³) consistently outperformed uniform quantization (QPruner¹) and simple MI-based allocation (QPruner²).

Conclusion

QPruner represents a significant step forward in LLM compression by intelligently combining pruning and quantization. By treating quantization as a probabilistic decision process, it navigates the complex trade-offs of model efficiency. While the Bayesian Optimization process introduces some computational overhead, the resulting memory savings and performance gains make it a compelling framework for deploying large-scale models on resource-constrained devices. Future work may explore ultra-low bit-widths to further push the boundaries of model efficiency.

Summary updated

Abstract

The rise of large language models (LLMs) has significantly advanced various natural language processing (NLP) tasks. However, the resource demands of these models pose substantial challenges. Structured pruning is an effective approach to reducing model size, but it often results in significant accuracy degradation, necessitating parameter updates to adapt. Unfortunately, such fine-tuning requires substantial memory, which limits its applicability. To address these challenges, we introduce quantization into the structured pruning framework to reduce memory consumption during both fine-tuning and inference. However, the combined errors from pruning and quantization increase the difficulty of fine-tuning, requiring a more refined quantization scheme. To this end, we propose QPruner, a novel framework that employs structured pruning to reduce model size, followed by a layer-wise mixed-precision quantization scheme. Quantization precisions are assigned to each layer based on their importance to the target task, and Bayesian optimization is employed to refine precision allocation strategies, ensuring a balance between model accuracy and memory efficiency. Extensive experiments on benchmark datasets demonstrate that QPruner significantly outperforms existing methods in memory savings while maintaining or improving model performance.
BibTeX
@inproceedings{zhou2025qpruner,
  author = {Changhai Zhou and Yuhua Zhou and Yibin Wang and Shijie Han and Qian Qiao and Hongguang Li},
  title = {{QPruner: Probabilistic Decision Quantization for Structured Pruning in Large Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.findings-naacl.240},
}
Figure