A 28nm 8-bit Floating-Point Tensor Core based CNN Training Processor with Dynamic Activation/Weight Sparsification

Shreyas Kolala Venkataramanaiah, Jian Meng, Han-Sok Suh, Injune Yeo, Jyotishman Saikia, Sai Kiran Cherupally, Yichi Zhang, Zhiru Zhang, Jae-sun SeoESSCIRC 2022 · OpenReview

Abstract

We present an 8-bit floating-point (FP8) training processor which implements (1) highly parallel tensor cores (fused multiply-add trees) that maintain high utilization throughout forward propagation (FP), backward propagation (BP), and weight update (WU) phases of the training process, (2) hardware-efficient channel gating for dynamic output activation sparsity, (3) dynamic weight sparsity based on group Lasso, and (4) gradient skipping based on FP prediction error. We develop a custom ISA to flexibly support different CNN topologies and training parameters. The 28nm prototype chip demonstrates large improvements in FLOPs reduction (7.3 ×), energy efficiency (16.4 TFLOPS/W), and overall training latency speedup (4.7×), for both supervised and self-supervised training tasks.
BibTeX
@inproceedings{venkataramanaiah2022nm,
  author = {Shreyas Kolala Venkataramanaiah and Jian Meng and Han-Sok Suh and Injune Yeo and Jyotishman Saikia and Sai Kiran Cherupally and Yichi Zhang and Zhiru Zhang and Jae-sun Seo},
  title = {{A 28nm 8-bit Floating-Point Tensor Core based CNN Training Processor with Dynamic Activation/Weight Sparsification}},
  booktitle = {ESSCIRC},
  year = {2022},
  url = {https://openreview.net/forum?id=IlWA6wl1wN},
}

Loading related content...

Figure