Deep Neural Network Acceleration With Sparse Prediction Layers

Zhongtian Yao, Kejie Huang, Haibin Shen, Zhaoyan MingIEEE Access 2020 · OpenReview

Abstract

The ever-increasing computation cost of Convolutional Neural Network (CNN) makes it imperative for real-world applications to accelerate the key steps especially the inference. In this work, we propose an efficient yet general scheme called Sparse Prediction Layer (SPL) which can predict and skip the trivial elements in the CNN layer. Pruned weights are used to predict the locations of maximum values in max-pooling kernels and those of positive values before Rectified Linear Units (ReLUs). Thereafter, the precise values of these predicted important elements are calculated selectively and the complete outputs are restored from them. Our experiments on ImageNet Large Scale Visual Recognition Competition (ILSVRC) 2012 show that SPL can reduce 68.3%, 58.6% and 59.5% Floating-point Operations (FLOPs) on AlexNet, VGG-16 and ResNet-50, respectively, within an accuracy loss of less than 1% without retraining. The proposed SPL scheme can further accelerate these networks pruned by other pruning-based methods, such as a FLOP reduction of 50.2% on the ResNet-50 which has been pruned by Channel Pruning (CP) before being applied with SPLs. A special matrix multiplication called Sparse Result Matrix Multiplication (SRMM) is proposed to support the implementation of SPL, and its acceleration effect is in line with expectations.

Introduction

Convolutional Neural Networks (CNNs) have revolutionized computer vision but come with significant computational costs, hindering deployment on resource-constrained devices. Traditional acceleration methods like weight pruning often require extensive retraining to maintain accuracy. This research introduces the Sparse Prediction Layer (SPL), a novel approach that accelerates inference by predicting and skipping redundant calculations. Instead of focusing solely on weights, SPL targets output redundancy—specifically, values that are discarded by activation functions like ReLU or pooling layers.

The Sparse Prediction Layer (SPL) Framework

The core of the SPL scheme is a two-stage inference process designed to identify "important" output elements before performing full-scale computation. This process is divided into four distinct phases:

  1. Pruning Phase: Weights are pruned to create a sparse set used for prediction.
  2. Location Prediction Phase: A "draft" convolution is performed using these sparse weights to predict the locations of significant outputs (e.g., positive values for ReLU or maximum values for pooling).
  3. Important Element Value Calculation: Precise values are calculated only for the predicted locations using the full weight set.
  4. Output Restoration Phase: The final output tensor is reconstructed by combining the precise values with zeros or placeholders for the skipped elements.
The four phases of the Sparse Prediction Layer (SPL) scheme, from weight pruning to final output restoration.

To ensure hardware efficiency, the authors developed Sparse Result Matrix Multiplication (SRMM). Unlike standard sparse operations that optimize for sparse inputs, SRMM is tailored to compute only a sparse subset of the output matrix, aligning theoretical FLOP reductions with actual execution speedups.

Implementation in Residual Networks

A critical insight of this work is that pruned weights, while insufficient for precise value estimation, are highly effective at predicting the location of important features. This is particularly true for max-pooling layers, where the "max" signal is robust against pruning.

For complex architectures like ResNet, the SPL logic is adapted to handle residual connections. In a bottleneck block, predictions are made for the initial 1x1 convolution and the combined output of the final 1x1 convolution and the shortcut path. This allows the network to skip computations even in layers with complex data flows.

SPL implementation within a ResNet bottleneck block, demonstrating how zero locations are predicted for different layers and shortcut paths.

Experimental Results

The SPL method was evaluated on the ImageNet (ILSVRC 2012) dataset across several standard architectures. Notably, the method requires no retraining, allowing it to be applied directly to pre-trained models. Key results include:

  • AlexNet: 68.3% FLOP reduction with <1% accuracy loss.
  • VGG-16: 58.6% FLOP reduction with <1% accuracy loss.
  • ResNet-50: 59.5% FLOP reduction with <1% accuracy loss.

Furthermore, SPL is complementary to existing pruning techniques. When applied to a ResNet-50 model already optimized via Channel Pruning (CP), SPL achieved an additional 50.2% reduction in FLOPs. The researchers also introduced the Susceptibility Ascending Order Scheme (SAOS), a greedy search algorithm to determine the optimal sparsity ratio for each layer, balancing efficiency and accuracy.

Conclusion

Sparse Prediction Layers offer a powerful, retraining-free method for accelerating CNN inference by exploiting output redundancy. By separating location prediction from value calculation and utilizing the SRMM kernel, SPL achieves significant practical speedups. While some layers are more sensitive to pruning than others, the proposed SAOS search helps mitigate accuracy loss. Future work may extend this approach to average-pooling layers or further optimize kernels for specialized AI hardware like mobile GPUs and NPUs.

Summary updated

Abstract

The ever-increasing computation cost of Convolutional Neural Network (CNN) makes it imperative for real-world applications to accelerate the key steps especially the inference. In this work, we propose an efficient yet general scheme called Sparse Prediction Layer (SPL) which can predict and skip the trivial elements in the CNN layer. Pruned weights are used to predict the locations of maximum values in max-pooling kernels and those of positive values before Rectified Linear Units (ReLUs). Thereafter, the precise values of these predicted important elements are calculated selectively and the complete outputs are restored from them. Our experiments on ImageNet Large Scale Visual Recognition Competition (ILSVRC) 2012 show that SPL can reduce 68.3%, 58.6% and 59.5% Floating-point Operations (FLOPs) on AlexNet, VGG-16 and ResNet-50, respectively, within an accuracy loss of less than 1% without retraining. The proposed SPL scheme can further accelerate these networks pruned by other pruning-based methods, such as a FLOP reduction of 50.2% on the ResNet-50 which has been pruned by Channel Pruning (CP) before being applied with SPLs. A special matrix multiplication called Sparse Result Matrix Multiplication (SRMM) is proposed to support the implementation of SPL, and its acceleration effect is in line with expectations.
BibTeX
@inproceedings{yao2020deep,
  author = {Zhongtian Yao and Kejie Huang and Haibin Shen and Zhaoyan Ming},
  title = {{Deep Neural Network Acceleration With Sparse Prediction Layers}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=SL0Zsf7hfs},
}
Figure