TL;DR
This paper introduces a novel "Multi-Perspective Early Exiting" (MPEE) framework to significantly speed up inference for large language models (LLMs) without sacrificing accuracy. By simultaneously considering both vertical (layer-wise) and horizontal (token-wise) perspectives for early exiting, MPEE achieves substantial acceleration while maintaining competitive performance.
What Problem Does This Paper Address?
Pretrained Language Models (PLMs) like BERT have revolutionized NLP, but their massive size leads to computationally expensive inference, making them impractical for resource-constrained devices or time-sensitive applications. Existing methods for accelerating inference, such as model compression (distillation, pruning, quantization) or traditional "early exiting" (EE) techniques, often result in a noticeable drop in performance. Traditional EE methods primarily focus on deciding which layers a sample should exit from (vertical perspective), ignoring the redundancy within tokens at each layer (horizontal perspective). This paper addresses the gap by proposing a unified approach that leverages both vertical and horizontal perspectives for more efficient and effective early exiting.
Methods
The core contribution is the Unified Multi-Perspective Early Exiting (MPEE) framework. It combines two orthogonal strategies:
- Sequential Token-wise Early Exiting (Horizontal Perspective): This method focuses on identifying and discarding less informative tokens within a layer. It uses "class attention" to gauge a token's contribution to the final prediction. To address the instability of class attention in shallow layers, it employs "recycling class attention memory." Less informative tokens are not entirely discarded but are fused into an "EE fusion token" to retain some information for subsequent layers.
- Layer-wise Early Exiting (Vertical Perspective): This is a more standard EE approach where EE classifiers are placed between transformer layers. Samples with high confidence can exit early, skipping computation in deeper layers. To improve the performance of these shallow EE classifiers, MPEE uses "recycling EE classifier memory" and "weighted ensemble self-distillation."
These two perspectives are unified to achieve both horizontal and vertical sparsity, leading to greater computational savings.
Datasets
The paper evaluates MPEE on the General Language Understanding Evaluation (GLUE) benchmark, which includes:
- SST-2: Sentiment analysis.
- QQP: Question pair similarity.
- MRPC: Paraphrase detection.
- QNLI: Question answering inference.
- MNLI: Natural language inference.
- RTE: Textual entailment.
Key Insights and Ideas
- Token-level redundancy is a significant untapped resource for acceleration: While layer-wise early exiting is common, ignoring the redundancy within tokens at each layer limits potential speedups. MPEE's horizontal perspective directly addresses this.
- Class attention is a good indicator of token informativeness: The paper shows that tokens with higher class attention scores contribute more to the final prediction, making them valuable for guiding early exiting.
- Memory mechanisms are crucial for shallow EE components: Both "recycling class attention memory" and "recycling EE classifier memory" are vital for improving the robustness and effectiveness of shallow-layer decisions.
- Orthogonal perspectives lead to synergistic gains: Combining horizontal and vertical early exiting is more effective than either alone, as they address different sources of redundancy.
Key Results
- Significant Acceleration: MPEE achieves substantial acceleration ratios across various GLUE tasks. For example, on SST-2, it reaches an acceleration ratio of 4.4x while maintaining 91.6% accuracy.
- Competitive Performance: MPEE generally outperforms baseline methods in terms of both acceleration and performance. It often achieves better accuracy than existing EE methods at comparable or higher acceleration ratios.
- Improved Tradeoff: Compared to methods like TR-BERT, which also reduces sequence length, MPEE offers a better performance-acceleration tradeoff. TR-BERT's RL-based approach can be less stable and requires retraining for different device platforms.
- Ablation Studies: Removing key components like EE fusion, attention memory, or classifier memory individually leads to a degradation in performance or acceleration, highlighting the importance of each module.
Limitations and Future Work
- Limitations: The paper acknowledges that while MPEE significantly reduces computation, it still requires some computation through the last layer for harder samples. The performance degradation, though small, is still present.
- Future Work: The authors suggest extending the proposed approach to vision and language pretrained models, considering the unique properties of different modalities.
Summary updated