Efficient Large-scale Transformer Training via Random and Layerwise Token Dropping
Lists
Only you see this
Abstract
Large-scale transformer models have become the de-facto architectures for various machine learning applications, e.g., CV and NLP. However, those large models also introduce prohibitive training costs. To mitigate this issue, we propose a novel random and layerwise token dropping method (\OURS), which skips the computation of a subset of the input tokens at all middle layers. Particularly, \OURS achieves considerable speedups and comparable accuracy as the standard training baseline. Compared to other token dropping methods, \OURS does not require (1) any importance score-based metrics, (2) any special token treatment (e.g., \texttt{[CLS]}), and (3) many layers in full sequence length training except the first and the last layers. Besides, a new \layertoken learning rate schedule is proposed for pretraining problems that resolve the heavy tuning requirement for our proposed training mechanism. Finally, we demonstrate that \OURS can be applied to broader applications, including \gpt and \bert pretraining as well as ViT and \gpt finetuning tasks. Our results show that \OURS can save about 33.3\% theoretical compute cost and 25.6\% wall-clock training time while achieving similar zero-shot evaluations on \gptb as compared to baseline.
BibTeX
@inproceedings{yao2023efficient,
author = {Zhewei Yao and Xiaoxia Wu and Conglong Li and Connor Holmes and Minjia Zhang and Cheng Li and Yuxiong He},
title = {{Efficient Large-scale Transformer Training via Random and Layerwise Token Dropping}},
booktitle = {International Conference on Learning Representations},
year = {2023},
url = {https://openreview.net/forum?id=CPg5IRu9PL},
}Loading related content...