Entropy Meets Importance: A Unified Head Importance–Entropy Score for Stable and Efficient Transformer Pruning

Minsik Choi, Hyegang Son, Changhoon Kim, Young Geun KimSubmission to ICLR 2026 · Desk rejected

TL;DR

Standard gradient-based head pruning causes Transformer performance to plummet under aggressive compression because it overlooks how attention heads distribute their focus across tokens. The authors propose the Head Importance–Entropy Score (HIES), which combines first-order loss sensitivity with attention entropy to protect functionally specialized, token-focused heads. Across language, vision, and multimodal models, HIES prevents catastrophic accuracy drops, boosting model quality by up to 15.2% and inference stability by $2.04\times$ over gradient-only baselines at high sparsity.

What Problem Does This Paper Address?

Deploying large Transformers on edge devices requires aggressive compression. Attention head pruning is an attractive structured compression technique because it cuts FLOPs and KV-cache memory usage without altering layer topology or checkpoint structures.

Most existing methods rank heads using a gradient-based Head Importance Score (HIS), which estimates the immediate impact of removing a head on task loss. However, HIS-pruned models exhibit a severe "sharp-drop" failure mode: once pruning exceeds a modest threshold (typically $\ge 30\%$), performance collapses.

The paper identifies the cause: HIS only measures output magnitude gradients, failing to recognize how heads allocate attention over input tokens. Consequently, a head that focuses sharply on a handful of decisive tokens (e.g., sentiment indicators) can register the same low HIS score as a head with diffuse, uninformative attention. Pruning these specialized heads destroys structural attention diversity and destroys inference stability under distribution shifts.

Methods

The authors introduce HIES (Head Importance–Entropy Score), a unified selection metric:

$$\text{HIES}_h = \alpha \widehat{\text{HIS}}_h + (1 - \alpha)(1 - \widetilde{\text{AE}}_h)$$

  • Min-Max Normalized Components: $\widehat{\text{HIS}}_h$ scales the standard first-order head importance score to $[0, 1]$. $\widetilde{\text{AE}}_h$ normalizes the token-averaged Attention Entropy (AE), where lower entropy reflects concentrated attention and higher entropy indicates uniform spreading.
  • Entropy Deficit as a Retention Signal: The term $(1 - \widetilde{\text{AE}}_h)$ penalizes heads with high entropy (diffuse attention) and retains low-entropy heads that focus tightly on critical tokens.
  • Theoretical Grounding:
  • Loss-Increase & Generalization Bounds: The authors prove that while HIS controls first-order training loss increases, AE deficit directly bounds the generalization gap via an entropy–total variation (TV) inequality.
  • Gradient Orthogonality: The gradients of HIS and AE with respect to attention probabilities are shown to be orthogonal in expectation. HIS captures contribution magnitude, while AE captures structural dispersion—confirming they provide non-redundant, complementary signals.

Datasets

The authors evaluate HIES across multiple modalities and architectures:

  • Natural Language Understanding (BERT-base): 8 tasks from the GLUE benchmark (SST-2, CoLA, MRPC, QQP, STS-B, QNLI, MNLI, RTE).
  • Large Language Models (LLaMA-2 7B): HellaSwag, Winogrande, ARC-Easy (ARC-e), ARC-Challenge (ARC-c), and OpenBookQA (OBQA).
  • Vision (ViT-Large): ImageNet-1k, CIFAR-100, Food-101, Fashion MNIST, and Oxford Flowers.
  • Multimodal (LLaVA-1.5 7B): VizWiz-VQA (visual questions asked by blind users) and MM-Vet (complex vision-language reasoning).

No new datasets were introduced; evaluations rely on standard open benchmarks.

Key Insights and Ideas

  • Two Distinct Pruning Regimes:
  • Redundancy Regime ($\le 10\%$ sparsity): The model contains abundant redundant heads. Here, simple HIS performs marginally better as it targets immediate, one-step loss sensitivity.
  • Specialization Regime ($\ge 30\%$ sparsity): Redundant heads are exhausted. HIS begins accidentally eliminating specialized heads, causing sharp degradation. HIES protects these specialized, low-entropy heads, maintaining stable performance.
  • Mitigating Bottom-Up Pruning Bias: HIS tends to prune lower Transformer layers disproportionately due to one-step backward pass dynamics. HIES distributes head pruning more evenly across lower, middle, and upper layers, maintaining cross-layer representational capacity.
  • Stability as a Robustness Proxy: Retaining low-entropy heads improves inference-time stability against input perturbations and pruning-induced distribution shifts.

Key Results

  • BERT on GLUE: At 50% pruning ratio, HIES achieves an average accuracy of 81.43% compared to 79.50% for HIS and 69.82% for random pruning, while improving prediction stability by up to 8.34% over HIS.
  • LLaMA-2 7B: Across 10%–60% sparsity, HIES consistently outperforms HIS, improving average zero-shot accuracy by up to +10.54% and stability by +6.21%, showing strong gains on complex reasoning benchmarks like ARC-c.
  • ViT-Large (Image Classification): At 30% sparsity, HIS experiences near-total failure (12.80% average accuracy; 6.20% on ImageNet-1k). In contrast, HIES sustains 47.70% average accuracy (55.40% on ImageNet-1k)—a 73.17% relative improvement over HIS.
  • LLaVA-1.5 7B (Multimodal Reasoning): At 50% pruning ratio, HIES yields an 11.71% relative gain over HIS on VizWiz-VQA and MM-Vet by protecting attention heads crucial for cross-modal alignment.

Limitations and Future Work

  • Hyperparameter Sensitivity: HIES introduces the mixing coefficient $\alpha$, which currently requires task-specific tuning on validation splits.
  • Scope of Pruning: The method currently focuses exclusively on attention heads. Extending entropy-aware criteria to Feed-Forward Network (FFN) channels, MLP blocks, or joint layer/head pruning pipelines remains unexplored.
  • Post-Pruning Recovery: The experiments evaluate post-pruning behavior without heavy retraining or distillation pipelines (e.g., LoRA recovery), leaving the interaction between HIES and parameter-efficient fine-tuning for future study.

Summary updated

Abstract

Transformer-based models have achieved remarkable performance in NLP tasks. However, their structural characteristics—multiple layers and attention heads—introduce efficiency challenges in inference and deployment. To address these challenges, various pruning methods have recently been proposed. Notably, gradient-based methods using Head Importance Scores (HIS) have gained traction for interpretability, efficiency, and ability to identify redundant heads. However, HIS alone has limitations as it captures only the gradient-driven contribution, overlooking the diversity of attention patterns. To overcome these limitations, we introduce a novel pruning criterion, **HIES (Head Importance-Entropy Score)**, which integrates head importance scores with attention entropy, providing complementary evidence on per-head contribution. Empirically, HIES‐based pruning yields up to 15.2\% improvement in model quality and $2.04\times$ improvement in stability over HIS‐only methods, enabling substantial model compression without sacrificing either accuracy or stability. Code will be released upon publication.
BibTeX
@inproceedings{choi2025entropy,
  author = {MINSIK CHOI and Hyegang Son and Changhoon Kim and Young Geun Kim},
  title = {{Entropy Meets Importance: A Unified Head Importance–Entropy Score for Stable and Efficient Transformer Pruning}},
  booktitle = {OPT @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=UKx7KhS9ZV},
}
Figure