On the Effectiveness of Adapting Pre-trained Transformer Models via Adversarial Noise
Lists
Only you see this
Abstract
Pretraining Transformer-based language models followed by adapting the pre-trained models to a downstream task is an effective transfer mechanism in NLP. While it is well-known that the pretraining stage is computationally expensive, the downstream adaptation also becomes costly as Transformers grow in size rapidly and the wide usage scenarios of fine-tuning pre-trained Transformers. In this work, we find that techniques that have demonstrated success in accelerating the pre-training tasks, such as large-batch optimizations, lead to severe accuracy degradation. We find strong regularization techniques such as adversarial training help to close the accuracy gap. However, the computational complexity associated with this approach, due to the high cost of generating adversaries, prevents it from reducing adaptation costs even with a large number of GPUs. As such, we systematically study both the computation efficiency and generalization of adversarial training for adapting pre-trained transformers, under a large-batch optimization regime. Our investigation yields simple yet effective algorithms for adapting transformer models. We show in experiments that our proposed method attains up to 9.8$\times$ adaptation speedups over the baseline on BERT$_{base}$ and RoBERTa$_{large}$, while achieving comparable and sometimes higher accuracy than fine-tuning using existing baselines.
BibTeX
@misc{zhang2022effectiveness,
author = {Minjia Zhang and Niranjan Hegde and Yuxiong He},
title = {{On the Effectiveness of Adapting Pre-trained Transformer Models via Adversarial Noise}},
year = {2022},
url = {https://openreview.net/forum?id=_5jWpg7TK9b},
howpublished = {OpenReview},
}Loading related content...