Introduction
Transformer-based encoders, such as BERT, have become foundational in Natural Language Processing (NLP). Despite their success, these models are significantly overparameterized, leading to high computational costs and latency during deployment. Existing pruning techniques often focus solely on attention heads, leaving the computationally expensive Feed-Forward Networks (FFNs) untouched. Furthermore, many methods rely on complex heuristic importance scores that do not always translate into tangible hardware speedups.
To address these challenges, we present SHRP (Specialized Head Routing and Pruning), a framework designed to compress Transformer encoders by treating attention heads as independent experts. SHRP enables simultaneous pruning of attention and FFN components, resulting in a static, high-performance architecture optimized for real-world deployment.
Methodology
The SHRP framework introduces a structural shift in the Transformer block, moving from a monolithic multi-head attention mechanism to a modular "Expert Attention" design.
Expert Attention and Expander FFN
In a standard Transformer, all attention heads are computed and concatenated before being processed by a large FFN. SHRP decouples this relationship. Each attention head is treated as an independent expert and is immediately followed by a shared, lightweight Expander FFN. This tight coupling allows the model to prune a head and its associated FFN logic as a single unit, facilitating much higher compression ratios than traditional methods.
Top-1 Gating and Specialization
During the training phase, a router selects exactly one head (Top-1) to process a given input. This "winner-takes-all" strategy forces individual heads to specialize in specific linguistic features. To ensure stable training and prevent model collapse, SHRP employs a progressive two-stage approach:
- Layer-wise Conversion: The model is converted to the SHRP format one layer at a time.
- Balanced Exploration: A load-balancing loss is initially used to encourage the model to utilize all available heads. As training progresses, this loss is removed, allowing the router to converge on the most efficient experts.
For final deployment, the dynamic router is removed, and only the most frequently utilized heads are retained, resulting in a deterministic, static model.
Experimental Results
SHRP was evaluated on the General Language Understanding Evaluation (GLUE) benchmark using BERT-base-uncased as the backbone.
Performance and Efficiency
The results demonstrate that SHRP achieves significant throughput gains with minimal impact on accuracy. In a moderate compression scenario (pruning 6 out of 12 layers), SHRP retained 93.1% of baseline accuracy while reducing parameters by 48%. Under extreme compression (retaining only 1 out of 12 layers), the model achieved a 4.2x throughput improvement and an 88% reduction in parameters, maintaining 84.4% accuracy.
Comparison with Baselines
SHRP consistently outperforms traditional pruning methods, particularly at high compression ratios. For instance, on the MNLI task with only 12 heads remaining, SHRP achieved an accuracy of 0.6853, significantly higher than the 0.4059 achieved by baseline methods like Joint DSP.
Conclusion
SHRP provides a robust framework for compressing Transformer encoders by jointly pruning attention heads and FFNs. By leveraging Top-1 gating and a progressive training strategy, SHRP produces static models that offer substantial hardware speedups without sacrificing significant accuracy. Future work will explore extending this methodology to autoregressive decoder models and investigating the benefits of keeping the dynamic router during inference.
Summary updated