DSC: Dense-Sparse Convolution for Vectorized Inference of Convolutional Neural Networks

Alexander Frickenstein, Manoj Rohit Vemparala, Christian Unger, Fatih Ayar, Walter StecheleWorkshops @ CVPR 2019 · DOI

Introduction

Deploying modern Convolutional Neural Networks (CNNs) in safety-critical environments, such as automotive systems, presents significant computational challenges. While model compression techniques like pruning and quantization are standard, they often introduce a "hardware-software gap." Modern CPUs utilize Single Instruction, Multiple Data (SIMD) units designed for regular, contiguous memory access. However, irregular sparsity—resulting from randomly deleted weights—leads to inefficient memory fetches and underutilized vector units. This paper introduces Dense-Sparse Convolution (DSC), a hybrid methodology designed to bridge this gap by balancing pruning regularity with vectorized hardware capabilities for low-latency inference.

Methodology

The proposed DSC framework optimizes CNN performance through a three-pronged strategy:

  1. Fixed-Point Quantization: The authors employ round-to-nearest fixed-point quantization (8-bit and 16-bit) to increase arithmetic intensity. This allows more operations per clock cycle and reduces the overall memory footprint.
  2. Winograd Convolution: For dense filters (those with low sparsity), the system uses the Winograd algorithm ($F(2\times2, 3\times3)$). Winograd transformations significantly reduce the number of required multiplications, making them highly efficient for small filter kernels.
  3. Dense-Sparse Convolution (DSC) Kernel: This core innovation categorizes filters based on a heuristic sparsity threshold. Sparse filters are processed using a direct sparse convolution kernel that utilizes the Compressed Sparse Row (CSR) format to skip zero-valued weights, while dense filters are handled by the optimized Winograd kernel. The results from both paths are accumulated to generate the final feature map.
Overview of the Dense-Sparse Convolution (DSC) methodology, illustrating the separation of sparse filters from dense filters for optimized processing.

Performance Analysis and Key Insights

A critical finding of this research is the existence of a "break-even" point where sparse kernels outperform dense Winograd kernels. This threshold is heavily influenced by data precision:

  • Sparsity Thresholds: In full-precision (32-bit) mode, sparse convolution only becomes faster than Winograd when sparsity exceeds 92%. However, at 16-bit precision, the break-even point drops to approximately 70% sparsity. Lower precision mitigates the memory bandwidth bottlenecks typically associated with irregular sparse access.
  • Feature Map Dynamics: Sparse convolution is more efficient on smaller feature maps (e.g., $28\times28$). On larger inputs (e.g., $224\times224$), the overhead of irregular memory access dominates, making dense algorithms preferable even in the presence of many zero weights.
Speedup relative to dense convolution across different pruning rates and precisions, highlighting the break-even points for sparse kernels.

Experimental Results

The methodology was evaluated using the VGG16 architecture on the ImageNet dataset across various platforms, including ARM Cortex A53 and Intel Xeon processors.

  • Speedup: On an Intel i5-6300, the DSC approach (using 16-bit precision and filter-wise pruning) achieved a latency of 174ms, representing a nearly 5x speedup over standard CPU implementations (858ms).
  • Accuracy: The optimized VGG16 model maintained a 68.5% Top-1 accuracy on ImageNet, effectively matching the performance of the original full-precision dense model.
  • Hardware Comparison: The software-based DSC implementation proved competitive with dedicated accelerators like FPGAs (Virtex-7) and mobile GPUs (Jetson TX1), demonstrating that optimized CPU kernels can provide a viable alternative to specialized hardware for real-time inference.

Conclusion and Future Work

DSC demonstrates that a hybrid approach to convolution can significantly accelerate pruned CNNs on general-purpose hardware. While current thresholding relies on heuristics, future work may involve automating threshold selection through formal optimization. Additionally, extending the DSC logic to accommodate the specific memory hierarchies of GPUs and exploring high-throughput batch processing remain promising avenues for further research.

Summary updated

BibTeX
@inproceedings{frickenstein2019dsc,
  author = {Alexander Frickenstein and Manoj Rohit Vemparala and Christian Unger and Fatih Ayar and Walter Stechele},
  title = {{DSC: Dense-Sparse Convolution for Vectorized Inference of Convolutional Neural Networks}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00175},
}
Figure