Spartan: A Sparsity-Adaptive Framework to Accelerate Deep Neural Network Training on GPUs
Lists
Only you see this
Abstract
Deep Neural Networks (DNNs) have emerged as an important class of machine learning algorithms, providing accurate solutions to a broad range of applications. Sparsity in activation maps in DNN training presents an opportunity to reduce computations. However, exploiting activation sparsity presents two major challenges: i) profiling activation sparsity during training comes with significant overhead due to computing the degree of sparsity and the data movement; ii) the dynamic nature of activation maps requires dynamic dense-to-sparse conversion during training, leading to significant overhead. In this article, we present Spartan, a lightweight hardware/software framework to accelerate DNN training on a GPU. Spartan provides a cost-effective and programmer-transparent microarchitectural solution to exploit activation sparsity detected during training. Spartan provides an efficient sparsity monitor, a tile-based sparse GEMM algorithm, and a novel compaction engine designed for GPU workloads. Spartan can reduce sparsity profiling overhead by 52.5× on average. For the most compute-intensive layers, i.e., convolutional layers, we can speedup AlexNet by 3.4×, VGGNet-16 by 2.14×, and ResNet-18 by 2.02×, when training on the ImageNet dataset.
BibTeX
@article{dong2021spartan,
author = {Shi Dong and Yifan Sun and Nícolas Bohm Agostini and Elmira Karimi and Daniel Lowell and Jing Zhou and José Cano and José Luis Abellán and David Kaeli},
title = {{Spartan: A Sparsity-Adaptive Framework to Accelerate Deep Neural Network Training on GPUs}},
journal = {IEEE Transactions on Parallel and Distributed Systems},
year = {2021},
doi = {10.1109/tpds.2021.3067825},
}