IG-Pruning: Input-Guided Block Pruning for Large Language Models

Ke Qiao, Shaolei Zhang, Yang Feng

TL;DR

This paper introduces IG-Pruning, a novel method for making large language models (LLMs) more efficient by dynamically pruning entire transformer blocks based on the input. Unlike static pruning methods, IG-Pruning adapts its pruning strategy at inference time, leading to significant performance gains and computational savings, especially in resource-constrained environments.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful but also computationally expensive, making them challenging to deploy, especially on devices with limited resources or for latency-sensitive applications. One promising approach to reduce this computational burden is "depth pruning," which involves removing entire transformer layers or blocks from the model. However, existing depth pruning methods typically rely on fixed pruning masks determined offline. This static approach is problematic because a single pruning strategy might not be optimal for all inputs or tasks, leading to performance degradation. The paper highlights that even masks achieving similar perplexity scores can exhibit vastly different performance on downstream tasks, indicating a need for more adaptive pruning.

Methods

IG-Pruning tackles this problem in two stages:

  1. Mask Candidate Discovery: This stage focuses on creating a diverse set of high-quality pruning masks.
  2. Input Clustering: The method first encodes input sentences from a calibration dataset into embeddings. It then uses K-means clustering to group these embeddings, creating clusters that represent semantically similar input patterns.
  3. Soft Mask Training: For each cluster, a dedicated layer mask is trained using a "hard concrete distribution" and Lo optimization. This process models transformer block masks (both attention and FFN blocks) and incorporates global information. The masks are initially "soft" (continuous values) during training, acting as scaling parameters, and are later "hardened" (binarized to 0 or 1) for inference. The training objective includes minimizing the LLM's language modeling loss for inputs within each cluster, along with a Lagrangian term to enforce a target sparsity.
  4. Dynamic Routing: This stage enables input-adaptive pruning at inference time.
  5. Embedding Pool and Routing: An embedding pool is maintained for each semantic cluster. When a new input arrives, its embedding is calculated, and its Euclidean distance to each cluster embedding is computed. The input is then assigned to the closest cluster.
  6. Mask Application: The pre-trained mask corresponding to the assigned cluster is directly applied to the model for that specific input, allowing selective skipping of layers. This mechanism requires no additional training of the base model parameters and incurs negligible routing overhead.

The paper also details how the dynamic routing handles FFN and Attention blocks differently. FFN layers are entirely skipped if their mask value is 0. For Attention layers, key and value projections are still computed to maintain the KV cache, but the computationally expensive scaled dot-product operation is bypassed if the mask value is 0.

Datasets

  • Calibration Data: The paper uses fineweb-edu (Lozhkov et al., 2024) for clustering and layer mask training. They also experimented with wikitext2 (Merity et al., 2016) and instruction datasets, finding fineweb-edu to yield superior results due to its high quality and diversity.
  • Evaluation Datasets: The models are evaluated on six zero-shot tasks using the lm-evaluation-harness (Gao et al., 2023):
  • OpenBookQA (Mihaylov et al., 2018)
    • Winogrande (Sakaguchi et al., 2021)
    • HellaSwag (Zellers et al., 2019)
    • PIQA (Bisk et al., 2020)
    • ARC-Easy and ARC-Challenge (Clark et al., 2018)

Key Insights and Ideas

  • Input-Adaptive Pruning is Crucial: The core insight is that static pruning masks are suboptimal. Dynamically adjusting pruning based on input semantics can significantly improve performance and efficiency.
  • Block-Level Pruning Outperforms Layer-Level: Experiments show that pruning individual blocks (attention and FFN) is more effective than pruning entire layers, offering greater flexibility.
  • Semantic Clustering Captures Input Nuances: Grouping inputs by semantic similarity allows the model to learn specialized pruning strategies for different types of inputs, leading to better overall performance.
  • High-Quality Calibration Data Matters: The choice of calibration data significantly impacts the effectiveness of the learned masks. Diverse, high-quality pretraining data like fineweb-edu leads to better results than more common datasets like wikitext2.
  • Attention vs. FFN Pruning Behavior: Analysis reveals that attention blocks, especially in middle layers, are more frequently pruned than FFN blocks, suggesting different roles and redundancies within the transformer architecture.

Key Results

  • Performance Gains: IG-Pruning consistently outperforms state-of-the-art static depth pruning methods across various sparsity levels and models.
  • For Llama-3-8B at 25% sparsity, IG-Pruning achieved 87.18% of dense model performance, surpassing the best baseline by 10.86 percentage points.
  • For Qwen-3-8B at 13.9% sparsity, IG-Pruning maintained 96.01% of dense performance, compared to 90.37% for the best baseline.
  • Computational Efficiency: At 25% sparsity, IG-Pruning reduces computational cost to 89.8% of the dense model. At 37.5% sparsity, this drops to 75.8%.
  • Model Generalization: The method shows strong performance across different LLM architectures (Llama-3, Qwen-3) and sizes, including smaller models like Llama-3.2-3B, Llama-3.2-1B, and Qwen-3-4B.
  • Number of Clusters: Increasing the number of clusters (up to 16) consistently improved performance, demonstrating the benefit of more specialized mask combinations.

Limitations and Future Work

  • Clustering Quality Dependency: The performance is sensitive to the quality of semantic clustering. If clusters are not effectively differentiated, performance might diminish.
  • Calibration Data Sensitivity: The method's performance is also sensitive to the quality and type of calibration data used.
  • Limited Task Scope: The evaluation primarily focused on specific zero-shot tasks. Generalization to other task types or domain-specific applications requires further validation.
  • Hyperparameter Sensitivity: The method introduces sensitivity to multiple hyperparameters, including Lo regularization, Lagrangian parameters, and the number of clusters.
  • Factual Consistency: The paper does not investigate the impact of block pruning on model factuality. Removing computational blocks could potentially affect factual recall and increase hallucinations.

Future Work Suggestions: * Combine dynamic pruning with hallucination mitigation techniques (e.g., TruthX, Truth-Aware Context Selection) to create models that are both efficient and factually reliable.

Summary updated

BibTeX
@misc{qiao2025ig,
  author = {Ke Qiao and Shaolei Zhang and Yang Feng},
  title = {{IG-Pruning: Input-Guided Block Pruning for Large Language Models}},
  year = {2025},
  eprint = {2511.02213},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.02213},
}
Figure