On the Impact of Calibration Data in Post-training Quantization and Pruning

Miles Williams, Nikolaos Aletras

TL;DR

This paper presents the first extensive empirical study on how calibration data affects post-training quantization and pruning in Large Language Models (LLMs). Contrary to the common assumption that these compression methods are robust to the choice of calibration data, the authors reveal that downstream task performance can vary significantly based on the specific random samples used. Based on these findings, they provide actionable recommendations to improve the reproducibility and effectiveness of LLM compression.

What Problem Does This Paper Address?

As Large Language Models scale up, their computational and memory demands skyrocket, making deployment in resource-constrained environments difficult. To tackle this, researchers use post-training compression (PTC) techniques like quantization (reducing weight precision) and pruning (removing redundant weights).

Unlike traditional compression that requires expensive retraining, PTC methods rely on a tiny "calibration dataset" (often just 128 random sentences) to estimate the distribution of internal layer activations and guide the compression process. Because it takes so few examples, the community has largely assumed that PTC is highly robust to the specific choice or distribution of calibration data. This paper challenges that assumption, systematically investigating whether the randomness in calibration data introduces hidden volatility into compressed model performance.

Methods

The authors conduct a massive empirical audit, creating 1,800 compressed models and running 19,800 total evaluations. They test two primary compression paradigms:

  1. Quantization: They evaluate GPTQ (a popular 4-bit weight quantization method) and SpQR (a near-lossless approach that keeps high-magnitude outlier weights in higher precision).
  2. Pruning: They evaluate SparseGPT and Wanda, both configured for 2:4 semi-structured sparsity (which zeroes out 2 out of every 4 weight elements to leverage GPU hardware acceleration).

As illustrated in the compression pipeline below, these methods ingest a pre-trained model and calibration data to output a streamlined, deployment-ready model.

Figure 1: Post-training compression methods rely upon calibration data to generate layer activations.

To isolate the impact of the calibration data, the authors hold all hyperparameters constant and vary only the calibration sets across different data sources, sampling 10 distinct, non-overlapping subsets of 128 examples for each source.

Datasets

The study draws calibration data from five diverse text corpora:

  • C4: A clean, massive crawl of English web text (the baseline).
  • CNN-DM: Long-form news articles.
  • RedPajama: An open-source replication of LLaMA's pre-training data.
  • RefinedWeb: A highly filtered, web-derived pre-training dataset.
  • Wikipedia: High-quality encyclopedic text.

The compressed models are evaluated across 11 standard NLP benchmarks assessing commonsense reasoning and language modeling: ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, LAMBADA, OpenBookQA, PIQA, RTE, StoryCloze, WinoGrande, and WikiText perplexity.

The experiments span three popular model families across multiple sizes: LLaMA (7B, 13B, 33B), Vicuna (7B, 13B, 33B), and OPT (6.7B, 13B, 30B).

Key Insights and Ideas

  • Random sampling introduces high performance variance: Simply changing the random seed used to select 128 calibration sentences from the exact same dataset can cause massive swings in downstream accuracy. For example, using SparseGPT on LLaMA-7B resulted in a 9% accuracy gap on the RTE task depending entirely on which C4 sample was chosen.
  • Pruning is far more sensitive than quantization: Pruning methods (especially SparseGPT) exhibit much higher performance dispersion across calibration sets than quantization methods. SpQR, because it isolates and protects outlier weights, proved to be the most robust to calibration data variance.
  • Perplexity is a deceptive metric: PTC papers traditionally use language modeling perplexity (e.g., on WikiText) to prove robustness. However, the authors show that a compressed model can display a stable, low perplexity while its zero-shot accuracy on downstream reasoning tasks swings wildly.
  • Data source quality matters for pruning: For pruning, curated pre-training data like RefinedWeb generally yields higher average downstream accuracy than encyclopedic text like Wikipedia.

Key Results

The distribution of zero-shot task accuracy across ten different calibration sets sampled from C4 highlights this volatility, particularly for pruning methods:

Figure 2: Distribution of accuracy across ten calibration sets sampled from C4 for the LLaMA family of models.

When looking at data volume, the authors found that while quantization methods plateau almost immediately, SparseGPT continuously benefits from larger calibration sets well beyond the standard 128 examples:

Figure 3: The perplexity on WikiText (L) and mean zero-shot accuracy (R) for LLaMA-7B with each compression method. We present the mean value and standard deviation (shaded) across ten calibration sets sampled from C4.

Limitations and Future Work

The primary limitation acknowledged by the authors is their exclusive focus on English-language models, tasks, and calibration data. While PTC algorithms are theoretically language-agnostic, linguistic nuances could alter data sensitivity.

For future work, the authors suggest exploring how the original pre-training protocol or instruction-tuning affects a model's downstream sensitivity to calibration data. They also recommend that future PTC research explicitly release their exact calibration data slices to ensure reproducibility, rather than relying on random seeds which can behave inconsistently across different software environments.

Summary updated

Abstract

Quantization and pruning form the foundation of compression for neural networks, enabling efficient inference for large language models (LLMs). Recently, various quantization and pruning techniques have demonstrated remarkable performance in a post-training setting. They rely upon calibration data, a small set of unlabeled examples that are used to generate layer activations. However, no prior work has systematically investigated how the calibration data impacts the effectiveness of model compression methods. In this paper, we present the first extensive empirical study on the effect of calibration data upon LLM performance. We trial a variety of quantization and pruning methods, datasets, tasks, and models. Surprisingly, we find substantial variations in downstream task performance, contrasting existing work that suggests a greater level of robustness to the calibration data. Finally, we make a series of recommendations for the effective use of calibration data in LLM quantization and pruning.
BibTeX
@inproceedings{williams2024impact,
  author = {Miles Williams and Nikolaos Aletras},
  title = {{On the Impact of Calibration Data in Post-training Quantization and Pruning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=F4PP8ar0w8},
}
Figure