Ultra-Low Precision 4-bit Training of Deep Neural Networks

Xiao Sun, Naigang Wang, Chia-Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Srinivasan, Kailash Gopalakrishnan

Abstract

In this paper, we propose a number of novel techniques and numerical representation formats that enable, for the very first time, the precision of training systems to be aggressively scaled from 8-bits to 4-bits. To enable this advance, we explore a novel adaptive Gradient Scaling technique (Gradscale) that addresses the challenges of insufficient range and resolution in quantized gradients as well as explores the impact of quantization errors observed during model training. We theoretically analyze the role of bias in gradient quantization and propose solutions that mitigate the impact of this bias on model convergence. Finally, we examine our techniques on a spectrum of deep learning models in computer vision, speech, and NLP. In combination with previously proposed solutions for 4-bit quantization of weight and activation tensors, 4-bit training shows a non-significant loss in accuracy across application domains while enabling significant hardware acceleration (> 7X over state-of-the-art FP16 systems).
BibTeX
@inproceedings{sun2020ultra,
  author = {Xiao Sun and Naigang Wang and Chia-Yu Chen and Jiamin Ni and Ankur Agrawal and Xiaodong Cui and Swagath Venkataramani and Kaoutar El Maghraoui and Vijayalakshmi Srinivasan and Kailash Gopalakrishnan},
  title = {{Ultra-Low Precision 4-bit Training of Deep Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
}
Figure