Abstract
Fine-tuning over large pretrained language models (PLMs) has established many state-of-the-art results. Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications. Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively that fine-tuning layers in a top-down manner is a promising solution. In this paper, we first point out that this method does not always work out due to the different convergence speeds of different layers/modules. Inspired by this observation, we propose a simple component-wise gradient norm clipping method to adjust the convergence speed for different components. Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability.
Introduction
Fine-tuning pretrained language models (PLMs) like BERT has become the standard paradigm for natural language processing tasks. However, this process is notoriously unstable, particularly on smaller datasets. Two training runs with identical hyperparameters but different random seeds can yield drastically different outcomes—one achieving state-of-the-art performance and another failing to converge entirely.
While previous research identified "catastrophic forgetting" as a culprit, suggesting that gradual unfreezing of layers could mitigate instability, this paper identifies a more fundamental issue: convergence racing.
The Convergence Racing Problem
The authors observe that different components of a model (e.g., attention matrices vs. feed-forward networks) learn at different speeds. If certain components converge to a local optimum too quickly, they can effectively "lock" the rest of the model, preventing it from reaching a better global solution. This explains why gradual unfreezing often underperforms; early-tuned parameters converge so rapidly that they cannot effectively coordinate with layers unfrozen later in the process.
By visualizing gradient norms, the researchers found that failed runs are characterized by extreme spikes and imbalances across layers, as seen in the figure above. In contrast, successful runs maintain balanced, well-bounded gradients.
Component-Wise Gradient Norm Clipping (CWGNC)
To address this imbalance, the authors propose Component-Wise Gradient Norm Clipping (CWGNC). Unlike standard gradient clipping, which scales the entire gradient vector based on a single global threshold, CWGNC operates at a granular level.
The method involves:
- Identifying individual "components" within the architecture (e.g., weight matrices for Key, Query, and Value in Transformers).
- Calculating and clipping the gradient norm for each component independently.
- Applying updates only after every component has been constrained by a uniform "speed limit."
This ensures that no single part of the model converges too quickly, allowing the entire architecture to adapt in harmony.
Experimental Results
The effectiveness of CWGNC was evaluated on the GLUE benchmark, focusing on tasks known for instability: RTE, MRPC, and CoLA, using BERT-large-uncased.
Performance and Stability
On the RTE task, standard fine-tuning exhibited a mean accuracy of 50.9 with a high standard deviation of 4.5. CWGNC significantly improved these metrics, achieving a mean accuracy of 73.1 and reducing the standard deviation to 1.3. Similar stability gains were observed in CoLA, where the standard deviation dropped from a volatile 25.6 to just 1.7.
Efficiency and Robustness
CWGNC also proved to be more efficient, reaching peak performance in just 5 epochs, compared to the 20 epochs required by other stable baselines. Furthermore, the method is robust to hyperparameter choices, performing consistently across a wide range of clipping thresholds (0.01 to 1.0).
Conclusion
Component-Wise Gradient Norm Clipping offers a simple yet powerful solution to the instability of PLM fine-tuning. By balancing the learning speeds of different model components, CWGNC ensures that fine-tuning is no longer a gamble, providing researchers and practitioners with a reliable path to high-performance models. Future work will explore the application of this method to even larger foundation models and a broader range of pretraining objectives.
Summary updated