TL;DR
This paper introduces Sensitivity-Based Layer Dropping (SBLD), a novel method to significantly speed up the training of large transformer models by intelligently dropping layers. By analyzing layer-wise sensitivity and employing a variance-based scheduler, SBLD achieves substantial throughput improvements (up to 35%) with minimal or even improved accuracy compared to existing methods.
What Problem Does This Paper Address?
Transformer models, the backbone of modern AI applications in NLP and Computer Vision, are notoriously computationally expensive to train. This massive computational burden, especially for large models with billions of parameters, makes pre-training and fine-tuning time-consuming and resource-intensive. While some methods have explored reducing this workload by skipping layers during training, they often rely on simple probability distributions that can lead to significant accuracy degradation. The challenge is to accelerate training without sacrificing model performance, particularly for the largest transformer architectures.
Methods
The core of SBLD lies in its two main components:
- Layer-Wise Sensitivity Analysis: Instead of uniform or depth-based layer dropping probabilities, SBLD first measures the "sensitivity" of each transformer layer. This is done by training the model for a few iterations with each layer individually dropped with a high probability. The resulting increase in loss for each layer indicates its importance. Layers with higher sensitivity are less likely to be dropped, while less sensitive layers are candidates for dropping.
- Block/Allow Lists: To further refine the dropping strategy and prevent accuracy loss, SBLD incorporates "Block" and "Allow" lists. Layers identified as highly sensitive (resulting in a large loss increase when dropped) are added to the Block list, meaning they are never dropped. Conversely, layers that show minimal loss increase or even improvement when dropped are added to the Allow list, making them more likely to be dropped.
- Variance-Based Scheduler: The paper argues that the impact of layer dropping changes throughout the training process. To account for this, SBLD uses a variance-based scheduler, specifically fitting the observed variance between baseline and dropped-layer training loss with a Poly-Weighted Exponential Gamma Distribution (PWEGD). This scheduler dynamically adjusts the overall keep probability for the entire model at each training iteration, aiming to optimize the trade-off between speed and accuracy as training progresses.
Datasets
The paper evaluates SBLD on several standard datasets and models:
- GPT-3: Experiments were conducted on GPT-3 models of varying sizes (1.7B and 3.6B parameters) for pre-training.
- LAMBADA: The pre-trained GPT-3 models were then evaluated on the LAMBADA dataset for a downstream task of predicting the last word of sentences.
- Swin Transformer V2-L: The method was also applied to the Swin Transformer V2-L model for pre-training on ImageNet-21k and fine-tuning on ImageNet-1k.
Key Insights and Ideas
- Layer Importance is Not Uniform: The most significant insight is that different layers in a transformer have vastly different impacts on model performance. Simply dropping layers based on depth or a fixed probability is suboptimal.
- Sensitivity Analysis is Key: Quantifying layer sensitivity provides a more principled way to decide which layers can be safely dropped or are even beneficial to drop.
- Dynamic Scheduling Matters: The optimal strategy for dropping layers isn't static; it evolves throughout training. A dynamic scheduler that adapts to the training stage is crucial for maintaining accuracy.
- Block/Allow Lists as a Safety Net: The inclusion of Block/Allow lists acts as a crucial safeguard, preventing the dropping of critical layers and ensuring that even layers that benefit from dropping are not dropped too aggressively.
- SBLD Outperforms PLD: The paper demonstrates that SBLD consistently outperforms the Progressive Layer Dropping (PLD) method, especially on larger models, by achieving better accuracy and/or faster training times.
Key Results
- GPT-3 (1.7B): SBLD improved throughput by up to 34.69% with minimal accuracy drop.
- GPT-3 (3.6B): SBLD improved throughput by up to 31.18% with minimal accuracy drop.
- GPT-3 Medium (0.35B) on LAMBADA: SBLD achieved higher accuracy (50.75% vs. 48.09%) with 8.06% less training time compared to PLD. In some cases, SBLD achieved a 35% speedup for the same accuracy.
- Swin Transformer V2-L: SBLD achieved better accuracy (87.74% vs. 87.60%) and faster convergence. It also reduced fine-tuning time by 35% with a negligible accuracy difference.
- Ablation Study: The variance-based scheduler (PWEGD) provided 3.8% extra throughput and 0.02 loss reduction, while the Block/Allow Lists feature contributed an additional 2.54% throughput improvement and 0.01 loss reduction.
Limitations and Future Work
- Dynamic Sensitivity Measurement: The authors suggest extending SBLD to other model architectures (CNNs, DNNs) by developing a dynamic sensitivity measurement method.
- Generalizability: While tested on transformers, the paper proposes exploring the efficiency of SBLD on other neural network architectures.
- Parameter Tuning: The paper mentions that the limit value $\theta$ for keep probability influences the trade-off between accuracy and throughput, suggesting further exploration of optimal parameter settings.
Summary updated