SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression

Zeli Su, Ziyin Zhang, Wenzheng Zhang, Zhou Liu, Guixian Xu, Wentao ZhangarXiv 2025 · DOI · arXiv

Introduction

Transformer-based encoders, such as BERT, have become foundational in Natural Language Processing (NLP). Despite their success, these models are significantly overparameterized, leading to high computational costs and latency during deployment. Existing pruning techniques often focus solely on attention heads, leaving the computationally expensive Feed-Forward Networks (FFNs) untouched. Furthermore, many methods rely on complex heuristic importance scores that do not always translate into tangible hardware speedups.

To address these challenges, we present SHRP (Specialized Head Routing and Pruning), a framework designed to compress Transformer encoders by treating attention heads as independent experts. SHRP enables simultaneous pruning of attention and FFN components, resulting in a static, high-performance architecture optimized for real-world deployment.

Methodology

The SHRP framework introduces a structural shift in the Transformer block, moving from a monolithic multi-head attention mechanism to a modular "Expert Attention" design.

Expert Attention and Expander FFN

In a standard Transformer, all attention heads are computed and concatenated before being processed by a large FFN. SHRP decouples this relationship. Each attention head is treated as an independent expert and is immediately followed by a shared, lightweight Expander FFN. This tight coupling allows the model to prune a head and its associated FFN logic as a single unit, facilitating much higher compression ratios than traditional methods.

Overview of the SHRP framework, illustrating the transition from standard Multi-Head Attention to Expert Attention and the subsequent pruning process.

Top-1 Gating and Specialization

During the training phase, a router selects exactly one head (Top-1) to process a given input. This "winner-takes-all" strategy forces individual heads to specialize in specific linguistic features. To ensure stable training and prevent model collapse, SHRP employs a progressive two-stage approach:

  1. Layer-wise Conversion: The model is converted to the SHRP format one layer at a time.
  2. Balanced Exploration: A load-balancing loss is initially used to encourage the model to utilize all available heads. As training progresses, this loss is removed, allowing the router to converge on the most efficient experts.

For final deployment, the dynamic router is removed, and only the most frequently utilized heads are retained, resulting in a deterministic, static model.

Experimental Results

SHRP was evaluated on the General Language Understanding Evaluation (GLUE) benchmark using BERT-base-uncased as the backbone.

Performance and Efficiency

The results demonstrate that SHRP achieves significant throughput gains with minimal impact on accuracy. In a moderate compression scenario (pruning 6 out of 12 layers), SHRP retained 93.1% of baseline accuracy while reducing parameters by 48%. Under extreme compression (retaining only 1 out of 12 layers), the model achieved a 4.2x throughput improvement and an 88% reduction in parameters, maintaining 84.4% accuracy.

Performance summary across all GLUE tasks, showing the trade-off between compression and accuracy.

Comparison with Baselines

SHRP consistently outperforms traditional pruning methods, particularly at high compression ratios. For instance, on the MNLI task with only 12 heads remaining, SHRP achieved an accuracy of 0.6853, significantly higher than the 0.4059 achieved by baseline methods like Joint DSP.

Conclusion

SHRP provides a robust framework for compressing Transformer encoders by jointly pruning attention heads and FFNs. By leveraging Top-1 gating and a progressive training strategy, SHRP produces static models that offer substantial hardware speedups without sacrificing significant accuracy. Future work will explore extending this methodology to autoregressive decoder models and investigating the benefits of keeping the dynamic router during inference.

Summary updated

Abstract

Transformer encoders are widely deployed in large-scale web services for natural language understanding tasks such as text classification, semantic retrieval, and content ranking. However, their high inference latency and memory consumption pose significant challenges for real-time serving and scalability. These limitations stem largely from architectural redundancy, particularly in the attention module. The inherent parameter redundancy of the attention mechanism, coupled with the fact that its attention heads operate with a degree of independence, makes it particularly amenable to structured model compression. In this paper, we propose SHRP (Specialized Head Routing and Pruning), a novel structured pruning framework that automatically identifies and removes redundant attention heads while preserving most of the model's accuracy and compatibility. SHRP introduces Expert Attention, a modular design that treats each attention head as an independent expert, followed by a lightweight shared expander feed-forward network that refines their outputs. The framework employs a unified Top-1 usage-driven mechanism to jointly perform dynamic routing during training and deterministic pruning at deployment. Experimental results on the GLUE benchmark using a BERT-base encoder show that SHRP achieves 93% of the original model accuracy while reducing parameters by 48 percent. Under an extreme compression scenario where 11/12 of the layers are pruned, the model still maintains 84% accuracy and delivers a 4.2x throughput gain while reducing computation to as low as 11.5 percent of the original FLOPs, demonstrating its practical utility for large-scale and latency-sensitive web deployments.
BibTeX
@misc{su2025shrp,
  author = {Zeli Su and Ziyin Zhang and Wenzheng Zhang and Zhou Liu and Guixian Xu and Wentao Zhang},
  title = {{SHRP: Specialized Head Routing and Pruning for Efficient Encoder Compression}},
  year = {2025},
  eprint = {2512.20635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.20635},
}
Figure