TL;DR
This paper presents the first extensive empirical study on how calibration data affects post-training quantization and pruning in Large Language Models (LLMs). Contrary to the common assumption that these compression methods are robust to the choice of calibration data, the authors reveal that downstream task performance can vary significantly based on the specific random samples used. Based on these findings, they provide actionable recommendations to improve the reproducibility and effectiveness of LLM compression.
What Problem Does This Paper Address?
As Large Language Models scale up, their computational and memory demands skyrocket, making deployment in resource-constrained environments difficult. To tackle this, researchers use post-training compression (PTC) techniques like quantization (reducing weight precision) and pruning (removing redundant weights).
Unlike traditional compression that requires expensive retraining, PTC methods rely on a tiny "calibration dataset" (often just 128 random sentences) to estimate the distribution of internal layer activations and guide the compression process. Because it takes so few examples, the community has largely assumed that PTC is highly robust to the specific choice or distribution of calibration data. This paper challenges that assumption, systematically investigating whether the randomness in calibration data introduces hidden volatility into compressed model performance.
Methods
The authors conduct a massive empirical audit, creating 1,800 compressed models and running 19,800 total evaluations. They test two primary compression paradigms:
- Quantization: They evaluate GPTQ (a popular 4-bit weight quantization method) and SpQR (a near-lossless approach that keeps high-magnitude outlier weights in higher precision).
- Pruning: They evaluate SparseGPT and Wanda, both configured for 2:4 semi-structured sparsity (which zeroes out 2 out of every 4 weight elements to leverage GPU hardware acceleration).
As illustrated in the compression pipeline below, these methods ingest a pre-trained model and calibration data to output a streamlined, deployment-ready model.
To isolate the impact of the calibration data, the authors hold all hyperparameters constant and vary only the calibration sets across different data sources, sampling 10 distinct, non-overlapping subsets of 128 examples for each source.
Datasets
The study draws calibration data from five diverse text corpora:
- C4: A clean, massive crawl of English web text (the baseline).
- CNN-DM: Long-form news articles.
- RedPajama: An open-source replication of LLaMA's pre-training data.
- RefinedWeb: A highly filtered, web-derived pre-training dataset.
- Wikipedia: High-quality encyclopedic text.
The compressed models are evaluated across 11 standard NLP benchmarks assessing commonsense reasoning and language modeling: ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, LAMBADA, OpenBookQA, PIQA, RTE, StoryCloze, WinoGrande, and WikiText perplexity.
The experiments span three popular model families across multiple sizes: LLaMA (7B, 13B, 33B), Vicuna (7B, 13B, 33B), and OPT (6.7B, 13B, 30B).
Key Insights and Ideas
- Random sampling introduces high performance variance: Simply changing the random seed used to select 128 calibration sentences from the exact same dataset can cause massive swings in downstream accuracy. For example, using SparseGPT on LLaMA-7B resulted in a 9% accuracy gap on the RTE task depending entirely on which C4 sample was chosen.
- Pruning is far more sensitive than quantization: Pruning methods (especially SparseGPT) exhibit much higher performance dispersion across calibration sets than quantization methods. SpQR, because it isolates and protects outlier weights, proved to be the most robust to calibration data variance.
- Perplexity is a deceptive metric: PTC papers traditionally use language modeling perplexity (e.g., on WikiText) to prove robustness. However, the authors show that a compressed model can display a stable, low perplexity while its zero-shot accuracy on downstream reasoning tasks swings wildly.
- Data source quality matters for pruning: For pruning, curated pre-training data like RefinedWeb generally yields higher average downstream accuracy than encyclopedic text like Wikipedia.
Key Results
The distribution of zero-shot task accuracy across ten different calibration sets sampled from C4 highlights this volatility, particularly for pruning methods:
When looking at data volume, the authors found that while quantization methods plateau almost immediately, SparseGPT continuously benefits from larger calibration sets well beyond the standard 128 examples:
Limitations and Future Work
The primary limitation acknowledged by the authors is their exclusive focus on English-language models, tasks, and calibration data. While PTC algorithms are theoretically language-agnostic, linguistic nuances could alter data sensitivity.
For future work, the authors suggest exploring how the original pre-training protocol or instruction-tuning affects a model's downstream sensitivity to calibration data. They also recommend that future PTC research explicitly release their exact calibration data slices to ensure reproducibility, rather than relying on random seeds which can behave inconsistently across different software environments.
Summary updated