TL;DR
Sequence-to-sequence models are powerful but computationally expensive. This paper shows that by strategically pruning only the decoder (asymmetric pruning), we can achieve significant inference speedups (up to 3x) with minimal loss in summarization accuracy. This asymmetry is key to optimizing for efficiency without sacrificing performance.
What Problem Does This Paper Address?
Large language models, particularly sequence-to-sequence (seq2seq) models used for tasks like abstractive summarization, are becoming increasingly powerful. However, their sheer size makes them difficult and expensive to deploy in real-world applications where latency is critical (e.g., on-device processing, real-time search results). The challenge is to make these models more efficient for inference without degrading their performance. While pruning is a common technique, simply pruning uniformly can lead to significant accuracy drops. This paper investigates how the asymmetry between the encoder and decoder in seq2seq models can be leveraged for more effective and efficient pruning.
Methods
The core of this work lies in structured pruning, which involves removing entire structural components (like layers or attention heads) rather than individual weights (unstructured pruning). The authors specifically explore asymmetric pruning, where they selectively prune either the encoder, the decoder, or both. They fine-tune the pruned models to recover performance. The key novelty is the systematic study of how pruning the encoder versus the decoder impacts both accuracy (measured by ROUGE-2 scores) and inference speed (latency). They also investigate how these effects change with model scale (from small to large FLAN-T5 models).
Datasets
The authors evaluate their methods on several widely used abstractive summarization datasets:
- CNN/DailyMail (CNNDM): A standard dataset for news summarization.
- XSUM: Known for its more abstractive and concise summaries.
- Query Independent Web Summary (QIWS): A proprietary dataset used to create informative contextual snippets for search engines.
Key Insights and Ideas
- Encoder for Accuracy, Decoder for Efficiency: A crucial insight is that encoder size is more strongly correlated with summarization accuracy, while decoder size is more directly linked to inference efficiency. This asymmetry is the foundation for their pruning strategy.
- Asymmetric Pruning is Key: Simply pruning the encoder or decoder alone, or pruning them symmetrically, doesn't yield the best results. The paper demonstrates that pruning only the decoder can lead to substantial speedups with minimal accuracy loss.
- Scale Matters for Compressibility: Larger models are more amenable to pruning. While smaller models experience significant performance degradation with pruning, larger models are more robust, allowing for greater compression without substantial accuracy drops.
- Consistency Across Datasets and Scales: The observed relationships between pruning, asymmetry, and scale were found to be remarkably consistent across different datasets and model sizes, suggesting a generalizable principle.
Key Results
- Significant Speedup: Asymmetric pruning, specifically targeting the decoder, achieved nearly 3x improvement in inference latency with only a ~1 point loss in ROUGE-2 score.
- Decoder Pruning Dominance: Pruning only the encoder resulted in a modest speedup (up to 30%) with a notable accuracy loss. In contrast, pruning only the decoder led to significant speedups (up to 400% in some cases) with much lower accuracy degradation.
- Scale Benefits: For example, on the QIWS dataset, pruning 50% of the layers on a small model caused an 80% drop in ROUGE-2, while the same compression on a larger model resulted in only a 40% drop.
- Uniform Pruning vs. Asymmetric: While uniform pruning can offer speedups, asymmetric pruning (especially decoder-only) provides a better trade-off between speed and accuracy.
Limitations and Future Work
- Batch Size Influence: The authors note that inference efficiency gains from asymmetric pruning are heavily influenced by batch size. While minimal batch sizes show significant differences, these gains become less pronounced as batch sizes increase.
- Focus on Structural Pruning: The study primarily focuses on structured pruning. The interaction of asymmetry with other compression techniques like quantization, unstructured pruning, and knowledge distillation is left for future work.
- Future Directions: The authors suggest exploring combinations of pseudo-labeling, early exiting, and quantization to further enhance inference efficiency in sequence-to-sequence models. They also plan to investigate the interplay of these techniques with asymmetry and scale.
Summary updated