TL;DR
Layer pruning—removing entire transformer blocks to compress LLMs—is surprisingly effective for classification tasks but devastating for generative reasoning. This paper demonstrates that pruning destroys "algorithmic circuits" responsible for basic arithmetic and code syntax, and while a novel fine-tuning strategy using Self-Generated Responses (SGR) helps, it cannot fully restore a model's reasoning depth.
What Problem Does This Paper Address?
As Large Language Models (LLMs) grow, the need for efficient compression becomes critical. Layer pruning is an attractive method because it’s simple: you just delete a few transformer blocks, reducing the model's depth and speeding up inference without needing complex hardware support.
Previous research suggested LLMs have significant "layer redundancy," showing that you could cut 20-25% of a model and still get high scores on multiple-choice benchmarks like MMLU. However, this paper identifies a major catch: these pruned models often "lose their minds" when asked to perform multi-step generative reasoning, such as solving math word problems or writing code. The authors set out to determine why this happens and whether the lost reasoning can be recovered under realistic, resource-constrained conditions.
Methods
The authors conducted a systematic study across several model families (Llama-3.1, Qwen-2.5, Mistral-v0.3, and Gemma-2). Their approach included:
- Single-Layer Ablation: They removed one layer at a time and tested the model to identify "load-bearing" layers for different tasks.
- Failure Mode Analysis: They moved beyond simple accuracy scores to look at how models failed, specifically testing "atomic" capabilities like basic arithmetic and parenthesis tracking in code.
- Self-Generated Responses (SGR): They proposed a simple recovery strategy. Instead of fine-tuning the pruned model on standard open-source datasets (which may have a distribution shift), they used the original, unpruned model to generate responses to prompts. They then used these "perfect" internal responses as targets to fine-tune the pruned model.
- Pruning Strategies: They compared standard methods like Block Influence (BI) and Reverse Order against a new Iterative greedy pruning approach.
Datasets
- Classification Benchmarks: HellaSwag, PIQA, MMLU, WinoGrande, OpenBookQA, ARC-E, and ARC-C.
- Generative Reasoning Benchmarks: GSM8K (math), HumanEval+ and MBPP+ (coding), and XSUM (summarization).
- Fine-tuning Datasets: Alpaca-cleaned and Dolci (used to generate the SGR training data).
Key Insights and Ideas
- Reasoning is Depth-Dependent: Unlike summarization, which remains robust even when layers are removed, math and coding are highly sensitive to depth. This suggests that "reasoning" isn't just about having the right knowledge; it requires a specific computational "circuit depth" to process information.
- Structural vs. Surface Failure: Pruning doesn't just cause "text degeneration" (rambling or looping). It induces a structural loss of capability. For example, a pruned model might correctly understand a math word problem but fail at the final step because it can no longer reliably calculate
32 x 6. - The "Parenthesis Specialist" Layers: The authors found that specific layers are critical for specific tasks. In Qwen-2.5, removing layer 23 caused a massive spike in "unbalanced parenthesis" errors in code, suggesting that certain layers act as specialized syntactic checkers.
- SGR as a "Healer": Fine-tuning on the model's own unpruned thoughts (SGR) outperformed standard supervised fine-tuning by 20–30 percentage points on generative tasks. This is likely because SGR provides targets that are already aligned with the model’s internal vocabulary and "style."
Key Results
- The Reasoning Gap: At a 25% pruning ratio, models retained ~90% of their classification performance but often crashed to ~15-30% of their original reasoning performance before fine-tuning.
- Recovery Limits: While SGR significantly improved results, it couldn't close the gap. For Llama-3.1-8B, SGR recovered reasoning to ~63% of the baseline, while classification performance was nearly fully restored (~91%).
- Arithmetic Degradation: Pruning dropped Llama’s arithmetic accuracy from 77% to 34%. Even after SGR fine-tuning, it only reached 58%, indicating that some "arithmetic circuits" are permanently broken when layers are removed.
- Pruning Ratios: The authors found that pruning is relatively "safe" for reasoning only at very low ratios (removing only ~2 layers). Beyond 10-15% pruning, the decline in reasoning becomes steep and difficult to reverse.
Limitations and Future Work
The study focused on "realistic" post-training—meaning they didn't use billions of tokens or massive compute for distillation. While massive-scale distillation might yield better recovery, it is often too expensive for most users.
The authors also noted that results are highly model-dependent; Qwen and Llama reacted differently to the same pruning strategies, suggesting that we need better ways to map "functional circuits" within LLMs before we start cutting them out. Future work could explore "circuit-aware" pruning that avoids removing these critical, non-redundant algorithmic blocks.
Summary updated