How Does Calibration Data Affect the Post-training Pruning and Quantization of Large Language Models?

Miles Williams, Nikolaos Aletras

Abstract

Pruning and quantization form the foundation of model compression for neural networks, enabling efficient inference for large language models (LLMs). Recently, various quantization and pruning techniques have demonstrated state-of-the-art performance in a post-training setting. They rely upon calibration data, a small set of unlabeled examples, to generate layer activations. However, no prior work has systematically investigated how the calibration data impacts the effectiveness of model compression methods. In this paper, we present the first extensive empirical study on the effect of calibration data upon LLM performance. We trial a variety of pruning and quantization methods, tasks, models, and datasets. Surprisingly, we find substantial variations in downstream task performance, contrasting existing work that suggests a greater level of robustness to the calibration data. Finally, we make a series of recommendations for the effective use of calibration data in LLM quantization and pruning.

Introduction

As Large Language Models (LLMs) scale to hundreds of billions of parameters, their deployment is increasingly hindered by massive memory and compute requirements. Post-training compression (PTC)—specifically quantization (reducing weight precision) and pruning (removing redundant weights)—has emerged as a popular solution because it avoids the prohibitive costs of model retraining.

PTC methods typically rely on a small "calibration set," often comprising just 128 unlabeled examples, to estimate the distribution of layer activations. Until now, the prevailing assumption in the research community was that these methods were largely "distribution-robust," meaning any random sample of web text would suffice. This study investigates whether this assumption holds or if the choice of calibration data is a critical, overlooked variable in the model compression pipeline.

Methodology

The authors conducted an extensive empirical study involving nine models across three families: LLaMA (7B, 13B, 33B), Vicuna (7B, 13B, 33B), and OPT (6.7B, 13B, 30B). They evaluated two quantization methods (GPTQ and SpQR) and two pruning methods (SparseGPT and Wanda).

The post-training compression pipeline: A pre-trained model and calibration data are used to produce quantized or pruned variants.

The study utilized five diverse data sources for calibration: C4 (Common Crawl), CNN-DM (news), RedPajama (LLaMA training set reproduction), RefinedWeb (filtered web content), and Wikipedia. To test for intra-dataset variability, ten distinct random samples were created for each source. Models were evaluated on 11 standard zero-shot NLP tasks (e.g., BoolQ, PIQA, HellaSwag) and WikiText perplexity, totaling nearly 20,000 individual evaluations.

Key Insights and Results

The "Robustness Myth" Debunked

Contrary to previous literature, the choice of calibration data matters significantly. For the same model and method, swapping one set of 128 random examples for another from the same source caused accuracy to fluctuate by as much as 9.4% on certain tasks. For instance, LLaMA-7B pruned with SparseGPT saw RTE task accuracy range from 52.7% to 61.7% based solely on the random calibration sample used.

Performance variability across zero-shot tasks for LLaMA models. The spread in violin plots indicates sensitivity to calibration data choice, with pruning methods (red/purple) showing significantly higher variance than quantization (blue).

Pruning Sensitivity vs. Quantization Stability

Pruning methods, particularly SparseGPT, exhibited much higher sensitivity to calibration data than quantization. The authors hypothesize that pruning is inherently more "destructive," making the accuracy of activation estimates derived from calibration data more critical. In contrast, SpQR (a near-lossless quantization method) proved to be the most robust, with mean accuracy variations typically around 0.2%.

Perplexity and Data Quality

A common practice is using perplexity as a proxy for compression quality. However, the study found that a "robust" perplexity score does not guarantee stable performance on downstream reasoning tasks. A model might maintain low perplexity but fail significantly on logic tasks like BoolQ. Regarding data sources, RefinedWeb generally emerged as the strongest calibration source, while Wikipedia often resulted in the poorest downstream performance, suggesting that diversity and "web-like" nature may be more important than formal structure.

Recommendations and Conclusion

The study concludes that while performance often plateaus after 128 calibration examples, the composition of those examples is paramount. The authors offer three practical recommendations:

  1. Release calibration data: Researchers should share the exact examples used to ensure reproducibility.
  2. Test multiple sets: Avoid relying on a single random sample during development.
  3. Manually inspect data: Given the small sample size, anomalous examples can disproportionately degrade the final model.

Future work should explore these effects in multilingual settings and at more aggressive compression levels, as this study focused on English-language models with 4-bit quantization and 50% 2:4 sparsity.

Summary updated

Abstract

Pruning and quantization form the foundation of model compression for neural networks, enabling efficient inference for large language models (LLMs). Recently, various quantization and pruning techniques have demonstrated state-of-the-art performance in a post-training setting. They rely upon calibration data, a small set of unlabeled examples, to generate layer activations. However, no prior work has systematically investigated how the calibration data impacts the effectiveness of model compression methods. In this paper, we present the first extensive empirical study on the effect of calibration data upon LLM performance. We trial a variety of pruning and quantization methods, tasks, models, and datasets. Surprisingly, we find substantial variations in downstream task performance, contrasting existing work that suggests a greater level of robustness to the calibration data. Finally, we make a series of recommendations for the effective use of calibration data in LLM quantization and pruning.
BibTeX
@inproceedings{williams2024how,
  author = {Miles Williams and Nikolaos Aletras},
  title = {{How Does Calibration Data Affect the Post-training Pruning and Quantization of Large Language Models?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=F4PP8ar0w8},
}
Figure