Improving Bert Fine-Tuning via Stabilizing Cross-Layer Mutual Information

Jicun Li, Xingjian Li, Tianyang Wang, Shi Wang, Yanan Cao, Cheng-Zhong Xu, Dejing Dou

Abstract

Fine-tuning pre-trained language models, such as BERT, has shown enormous success among various NLP tasks. Though simple and effective, the process of fine-tuning has been found unstable, which often leads to unexpected poor performance. To increase stability and generalizability, most existing works resort to maintaining the parameters or representations of pre-trained models during fine-tuning. Nevertheless, very little work explores mining the reliable part of pre-learned information that can help to stabilize fine-tuning. To address this challenge, we introduce a novel solution in which we fine-tune BERT with stabilized cross-layer mutual information. Our method aims to preserve the reliable behaviors of cross-layer information propagation, instead of preserving the information itself, of the pre-trained model. Therefore, our method circumvents the domain conflicts between pre-trained and target tasks. We conduct extensive experiments with popular pre-trained BERT variants on NLP datasets, demonstrating the universal effectiveness and robustness of our method.
BibTeX
@inproceedings{li2023improving,
  author = {Jicun Li and Xingjian Li and Tianyang Wang and Shi Wang and Yanan Cao and Cheng-Zhong Xu and Dejing Dou},
  title = {{Improving Bert Fine-Tuning via Stabilizing Cross-Layer Mutual Information}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10095747},
}

Loading related content...

Figure