On the Limits of Layer Pruning for Generative Reasoning in LLMs

Safal Shrestha, Anima Shrestha, Aadim Nepal, Minwu Kim, Zijian ZhouarXiv 2026 · DOI · arXiv

TL;DR

Layer pruning—removing entire transformer blocks to compress LLMs—is surprisingly effective for classification tasks but devastating for generative reasoning. This paper demonstrates that pruning destroys "algorithmic circuits" responsible for basic arithmetic and code syntax, and while a novel fine-tuning strategy using Self-Generated Responses (SGR) helps, it cannot fully restore a model's reasoning depth.

What Problem Does This Paper Address?

As Large Language Models (LLMs) grow, the need for efficient compression becomes critical. Layer pruning is an attractive method because it’s simple: you just delete a few transformer blocks, reducing the model's depth and speeding up inference without needing complex hardware support.

Previous research suggested LLMs have significant "layer redundancy," showing that you could cut 20-25% of a model and still get high scores on multiple-choice benchmarks like MMLU. However, this paper identifies a major catch: these pruned models often "lose their minds" when asked to perform multi-step generative reasoning, such as solving math word problems or writing code. The authors set out to determine why this happens and whether the lost reasoning can be recovered under realistic, resource-constrained conditions.

Methods

The authors conducted a systematic study across several model families (Llama-3.1, Qwen-2.5, Mistral-v0.3, and Gemma-2). Their approach included:

  1. Single-Layer Ablation: They removed one layer at a time and tested the model to identify "load-bearing" layers for different tasks.
  2. Failure Mode Analysis: They moved beyond simple accuracy scores to look at how models failed, specifically testing "atomic" capabilities like basic arithmetic and parenthesis tracking in code.
  3. Self-Generated Responses (SGR): They proposed a simple recovery strategy. Instead of fine-tuning the pruned model on standard open-source datasets (which may have a distribution shift), they used the original, unpruned model to generate responses to prompts. They then used these "perfect" internal responses as targets to fine-tune the pruned model.
  4. Pruning Strategies: They compared standard methods like Block Influence (BI) and Reverse Order against a new Iterative greedy pruning approach.

Datasets

  • Classification Benchmarks: HellaSwag, PIQA, MMLU, WinoGrande, OpenBookQA, ARC-E, and ARC-C.
  • Generative Reasoning Benchmarks: GSM8K (math), HumanEval+ and MBPP+ (coding), and XSUM (summarization).
  • Fine-tuning Datasets: Alpaca-cleaned and Dolci (used to generate the SGR training data).

Key Insights and Ideas

  • Reasoning is Depth-Dependent: Unlike summarization, which remains robust even when layers are removed, math and coding are highly sensitive to depth. This suggests that "reasoning" isn't just about having the right knowledge; it requires a specific computational "circuit depth" to process information.
  • Structural vs. Surface Failure: Pruning doesn't just cause "text degeneration" (rambling or looping). It induces a structural loss of capability. For example, a pruned model might correctly understand a math word problem but fail at the final step because it can no longer reliably calculate 32 x 6.
  • The "Parenthesis Specialist" Layers: The authors found that specific layers are critical for specific tasks. In Qwen-2.5, removing layer 23 caused a massive spike in "unbalanced parenthesis" errors in code, suggesting that certain layers act as specialized syntactic checkers.
  • SGR as a "Healer": Fine-tuning on the model's own unpruned thoughts (SGR) outperformed standard supervised fine-tuning by 20–30 percentage points on generative tasks. This is likely because SGR provides targets that are already aligned with the model’s internal vocabulary and "style."

Key Results

  • The Reasoning Gap: At a 25% pruning ratio, models retained ~90% of their classification performance but often crashed to ~15-30% of their original reasoning performance before fine-tuning.
  • Recovery Limits: While SGR significantly improved results, it couldn't close the gap. For Llama-3.1-8B, SGR recovered reasoning to ~63% of the baseline, while classification performance was nearly fully restored (~91%).
  • Arithmetic Degradation: Pruning dropped Llama’s arithmetic accuracy from 77% to 34%. Even after SGR fine-tuning, it only reached 58%, indicating that some "arithmetic circuits" are permanently broken when layers are removed.
  • Pruning Ratios: The authors found that pruning is relatively "safe" for reasoning only at very low ratios (removing only ~2 layers). Beyond 10-15% pruning, the decline in reasoning becomes steep and difficult to reverse.

Limitations and Future Work

The study focused on "realistic" post-training—meaning they didn't use billions of tokens or massive compute for distillation. While massive-scale distillation might yield better recovery, it is often too expensive for most users.

The authors also noted that results are highly model-dependent; Qwen and Llama reacted differently to the same pruning strategies, suggesting that we need better ways to map "functional circuits" within LLMs before we start cutting them out. Future work could explore "circuit-aware" pruning that avoids removing these critical, non-redundant algorithmic blocks.

Summary updated

Abstract

Recent works have shown that layer pruning can compress large language models (LLMs) while retaining strong performance on classification benchmarks with little or no finetuning. However, existing pruning techniques often suffer severe degradation on generative reasoning tasks. Through a systematic study across multiple model families, we find that tasks requiring multi-step reasoning are particularly sensitive to depth reduction. Beyond surface-level text degeneration, we observe degradation of critical algorithmic capabilities, including arithmetic computation for mathematical reasoning and balanced parenthesis generation for code synthesis. Under realistic post-training constraints, without access to pretraining-scale data or compute, we evaluate a simple mitigation strategy based on supervised finetuning with Self-Generated Responses. This approach achieves strong recovery on classification tasks, retaining up to 90\% of baseline performance, and yields substantial gains of up to 20--30 percentage points on generative benchmarks compared to prior post-pruning techniques. Crucially, despite these gains, recovery for generative reasoning remains fundamentally limited relative to classification tasks and is viable primarily at lower pruning ratios. Overall, we characterize the practical limits of layer pruning for generative reasoning and provide guidance on when depth reduction can be applied effectively under constrained post-training regimes.
BibTeX
@misc{shrestha2026limits,
  author = {Safal Shrestha and Anima Shrestha and Aadim Nepal and Minwu Kim and Zijian Zhou},
  title = {{On the Limits of Layer Pruning for Generative Reasoning in LLMs}},
  year = {2026},
  eprint = {2602.01997},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.01997},
}
Figure