Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping

Chenghao Yang, Xuezhe Ma

Abstract

Fine-tuning over large pretrained language models (PLMs) has established many state-of-the-art results. Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications. Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively that fine-tuning layers in a top-down manner is a promising solution. In this paper, we first point out that this method does not always work out due to the different convergence speeds of different layers/modules. Inspired by this observation, we propose a simple component-wise gradient norm clipping method to adjust the convergence speed for different components. Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability.

Introduction

Fine-tuning pretrained language models (PLMs) like BERT has become the standard paradigm for natural language processing tasks. However, this process is notoriously unstable, particularly on smaller datasets. Two training runs with identical hyperparameters but different random seeds can yield drastically different outcomes—one achieving state-of-the-art performance and another failing to converge entirely.

While previous research identified "catastrophic forgetting" as a culprit, suggesting that gradual unfreezing of layers could mitigate instability, this paper identifies a more fundamental issue: convergence racing.

The Convergence Racing Problem

The authors observe that different components of a model (e.g., attention matrices vs. feed-forward networks) learn at different speeds. If certain components converge to a local optimum too quickly, they can effectively "lock" the rest of the model, preventing it from reaching a better global solution. This explains why gradual unfreezing often underperforms; early-tuned parameters converge so rapidly that they cannot effectively coordinate with layers unfrozen later in the process.

Visualization of unstable gradient norms in a typical failed fine-tuning run, showing wild spikes and imbalances.

By visualizing gradient norms, the researchers found that failed runs are characterized by extreme spikes and imbalances across layers, as seen in the figure above. In contrast, successful runs maintain balanced, well-bounded gradients.

Component-Wise Gradient Norm Clipping (CWGNC)

To address this imbalance, the authors propose Component-Wise Gradient Norm Clipping (CWGNC). Unlike standard gradient clipping, which scales the entire gradient vector based on a single global threshold, CWGNC operates at a granular level.

The method involves:

  1. Identifying individual "components" within the architecture (e.g., weight matrices for Key, Query, and Value in Transformers).
  2. Calculating and clipping the gradient norm for each component independently.
  3. Applying updates only after every component has been constrained by a uniform "speed limit."

This ensures that no single part of the model converges too quickly, allowing the entire architecture to adapt in harmony.

Experimental Results

The effectiveness of CWGNC was evaluated on the GLUE benchmark, focusing on tasks known for instability: RTE, MRPC, and CoLA, using BERT-large-uncased.

Performance and Stability

On the RTE task, standard fine-tuning exhibited a mean accuracy of 50.9 with a high standard deviation of 4.5. CWGNC significantly improved these metrics, achieving a mean accuracy of 73.1 and reducing the standard deviation to 1.3. Similar stability gains were observed in CoLA, where the standard deviation dropped from a volatile 25.6 to just 1.7.

RTE Fine-Tuning Performance Change Across Time. CWGNC (red dashed line) shows superior and more stable performance compared to gradual unfreezing and baselines.

Efficiency and Robustness

CWGNC also proved to be more efficient, reaching peak performance in just 5 epochs, compared to the 20 epochs required by other stable baselines. Furthermore, the method is robust to hyperparameter choices, performing consistently across a wide range of clipping thresholds (0.01 to 1.0).

Conclusion

Component-Wise Gradient Norm Clipping offers a simple yet powerful solution to the instability of PLM fine-tuning. By balancing the learning speeds of different model components, CWGNC ensures that fine-tuning is no longer a gamble, providing researchers and practitioners with a reliable path to high-performance models. Future work will explore the application of this method to even larger foundation models and a broader range of pretraining objectives.

Summary updated

Abstract

Fine-tuning over large pretrained language models (PLMs) has established many state-of-the-art results. Despite its superior performance, such fine-tuning can be unstable, resulting in significant variance in performance and potential risks for practical applications. Previous works have attributed such instability to the catastrophic forgetting problem in the top layers of PLMs, which indicates iteratively that fine-tuning layers in a top-down manner is a promising solution. In this paper, we first point out that this method does not always work out due to the different convergence speeds of different layers/modules. Inspired by this observation, we propose a simple component-wise gradient norm clipping method to adjust the convergence speed for different components. Experiment results demonstrate that our method achieves consistent improvements in terms of generalization performance, convergence speed, and training stability. The codebase can be found at https://github.com/yangalan123/FineTuningStability.
BibTeX
@inproceedings{yang2022improving,
  author = {Chenghao Yang and Xuezhe Ma},
  title = {{Improving Stability of Fine-Tuning Pretrained Language Models via Component-Wise Gradient Norm Clipping}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.322},
}
Figure