TL;DR
When you compress a large language model (LLM) through pruning or quantization, it doesn't lose capabilities at random. Instead, capabilities fail in a remarkably consistent and predictable order across different models and compression methods, driven by how "securely" a model holds a task, how common the facts are, and how much output the format requires.
What Problem Does This Paper Address?
As we move LLMs from research labs to edge devices, compression—via pruning and quantization—is standard practice. While these techniques often preserve a model's average accuracy, they can cause specific capabilities to collapse entirely.
Until now, researchers disagreed on what fails first. Some suggested reasoning degrades before knowledge; others found the opposite. The authors of this paper argue that these disagreements stem from inconsistent evaluation (e.g., comparing a multiple-choice task to a generative one). They set out to determine if there is a universal "fragility profile" for LLMs and what factors actually dictate the order of failure.
Methods
The researchers conducted a massive, preregistered study applying seven types of "damage" to eight different instruction-tuned models (including the Qwen2.5 family, Llama-3.2, Mistral, Gemma-2, and Phi-3.5).
The damages included:
- Four random perturbations: Gaussian noise, weight zeroing, neuron ablation, and random layer removal.
- Three compression methods: Wanda pruning, SparseGPT pruning, and GPTQ quantization.
They evaluated 16 short-answer tasks across four different answer formats: Choose (multiple choice), Answer (bare string), JSON, and Twice (repeating the answer). To analyze the decline, they used a "rank-one model" where damage acts like a single dial being turned, and each capability fails at its own specific rate, defined by a "fragility exponent."
Datasets
The study utilized 16 diverse tasks, including:
- Generated tasks: Arithmetic (addition), naming capitals, key-code lookup, and word copying.
- Standard benchmarks: SciQ, BoolQ, ARC-Challenge, CommonsenseQA, MMLU (math and recall subsets), HellaSwag, PIQA, WinoGrande, and OpenBookQA.
- Frequency Analysis: To measure "fact familiarity," the authors used the Dolma v1.7 corpus (2.6 trillion tokens) to count the occurrences of specific answers in pretraining data.
Key Insights and Ideas
- The Shared Order: Within a model, different types of damage—even random noise—cause capabilities to fail in nearly the same order (median correlation 0.90). This suggests that fragility is a property of the model's internal knowledge structure, not the specific compression algorithm.
- The Three Axes of Fragility: The order of failure is determined by three separable factors:
- Task Security: Measured by the "answer margin" (the log-probability gap between the right answer and the best distractor). Secure tasks fail last.
- Fact Familiarity: Rare facts (low frequency in training data) are lost long before common ones.
- Answer Format: The more a model has to produce, the more fragile the capability. "Choose" is the most robust; "Twice" is the most fragile.
- The Semantic Dementia Parallel: The way LLMs fail mirrors human semantic dementia: losing rare knowledge before familiar knowledge, and recognition (choosing) before naming (producing).
- Predictability: Because these axes are shared, you can measure a model's fragility profile under random noise and use it to accurately predict how it will perform after actual compression (Wanda or SparseGPT).
Key Results
- High Correlation: Task effects under Wanda, SparseGPT, and GPTQ correlated with random-damage effects at a median of 0.875.
- Accuracy Prediction: Using a fragility profile measured under random noise, the authors predicted held-out task accuracy after compression with a median absolute error of 0.11–0.12. This is significantly better than assuming a uniform decline (0.20–0.21).
- Cross-Model Transfer: Fragility profiles even transferred across different model families. A profile measured on a Llama model could predict the failure order of a Qwen model almost as well as the Qwen model's own profile.
- Format Impact: Producing an answer in a specific template (like JSON) makes a task significantly more fragile. In many cases, the model still "knows" the answer but fails to follow the formatting template.
Limitations and Future Work
- Long-form Generation: The predictable order held for short-answer tasks but was less consistent for long-form tasks like multi-step reasoning, coding, or complex instruction following.
- GPTQ Outlier: While GPTQ followed the predicted order within a model, the order did not transfer well across models for this specific quantization method.
- The "Twice" Mystery: The authors found that asking a model to write an answer twice was the most fragile format, but they couldn't fully explain why this specific repetition is so difficult for damaged models.
- Deployment Safety: The study highlights that aggregate scores hide the loss of the "long tail" of knowledge. For safety-critical applications (like medical diagnosis of rare diseases), developers must specifically test for these rare events before deploying compressed models.
Summary updated