V-SKP: Vectorized Kernel-Based Structured Kernel Pruning for Accelerating Deep Convolutional Neural Networks

Kwanghyun Koo, Hyun Kim

Introduction

Deep Convolutional Neural Networks (DCNNs) have revolutionized computer vision but remain computationally expensive for edge deployment. Pruning—the removal of redundant parameters—is a standard solution, yet it faces a persistent trade-off. Unstructured pruning (weight pruning) maintains high accuracy but results in sparse matrices that GPUs cannot process efficiently. Conversely, structured pruning (filter pruning) is hardware-friendly but often degrades accuracy significantly and breaks architectural constraints like "shortcut" connections in ResNet.

V-SKP (Vectorized Kernel-based Structured Kernel Pruning) introduces a middle ground. By treating 2D kernels within a filter as the fundamental unit of pruning and employing a novel "kernel index convolution," V-SKP achieves significant real-world speedups on GPUs without the accuracy penalties of traditional structured methods.

The V-SKP Methodology

The core innovation of V-SKP lies in its granularity. While filter pruning removes entire 3D volumes, V-SKP removes specific 2D slices (kernels) within those volumes. This allows for a more surgical reduction of the network's parameters while maintaining a structure that can still be accelerated.

Comparison of (a) Filter pruning and (b) Kernel pruning granularities.

To make this approach effective, the authors introduced three key technical components:

  • Vectorized Importance Scoring: Traditional pruning often relies solely on weight magnitude (L1 norm). V-SKP treats each kernel as a vector and calculates importance based on both magnitude and direction (using cosine similarity). This ensures that kernels capturing unique feature patterns are preserved, even if their absolute weights are small.
  • Kernel Index Convolution: Standard convolution operations expect a dense mapping between input channels and filters. To avoid the overhead of processing "zeroed-out" kernels, the authors developed a custom CUDA-based convolution. This operation uses a Kernel Index Set to map input channels directly to the remaining active kernels, enabling true hardware acceleration.
  • Binary Index Compression: To ensure the metadata required for indexing doesn't offset storage gains, V-SKP stores indices as a bitmask. This "binary index" reduces memory overhead by up to 71.4% on large-scale datasets like ImageNet.

Experimental Results

The researchers evaluated V-SKP on CIFAR-10 and ImageNet (ILSVRC-2012) using ResNet architectures. The results demonstrate that V-SKP consistently outperforms state-of-the-art (SOTA) methods in the accuracy-vs-speed trade-off:

  • Real-World Acceleration: On a TITAN Xp GPU, V-SKP achieved a 2.72x speedup for ResNet-50 on ImageNet. This outperforms other structured methods like C-SGD, which reached 2.07x.
  • High Efficiency: For ResNet-50, the framework reduced parameters by 79.5% and FLOPs by 67.1%, with a negligible 0.90% drop in Top-1 accuracy.
  • Comparison to Unstructured Methods: V-SKP significantly outperformed KSE (a leading unstructured kernel pruning method), achieving 20.2% higher parameter reduction while maintaining superior accuracy.

Conclusion and Future Directions

V-SKP successfully bridges the gap between flexibility and hardware efficiency. By focusing on kernel-level granularity and optimizing the underlying CUDA operations, it allows deep networks to be compressed without sacrificing the structural regularity required by modern GPUs.

Future work includes further optimizing the custom CUDA kernels to match the performance of highly-tuned libraries like cuDNN and extending the structured kernel pruning paradigm to Vision Transformers (ViTs). Additionally, while the vectorized approach is most effective for kernels larger than 1x1, refining techniques for 1x1 layers remains an area for ongoing research.

Summary updated

Abstract

In recent years, kernel pruning, which offers the advantages of both weight and filter pruning methods, has been actively conducted. Although kernel pruning must be implemented as structured pruning to obtain the actual network acceleration effect on GPUs, most existing methods are limited in that they have been implemented as unstructured pruning. To compensate for this problem, we propose vectorized kernel-based structured kernel pruning (V-SKP), which has a high FLOPs reduction effect with minimal reduction in accuracy while maintaining a 4D weight structure. V-SKP treats the kernel of the convolution layer as a vector and performs pruning by extracting the feature from each filter vector of the convolution layer. Conventional L1/L2 norm-based pruning considers only the size of the vector and removes parameters without considering the direction of the vector, whereas V-SKP removes the kernel by considering both the feature of the filter vector and the size and direction of the vectorized kernel. Moreover, because the kernel-pruned weight cannot be utilized when using the typical convolution, in this study, the kernel-pruned weights and the input channels are matched by compressing and storing the retained kernel index in the kernel index set during the proposed kernel pruning scheme. In addition, a kernel index convolution method is proposed to perform convolution operations by matching the input channels with the kernel-pruned weights on the GPU structure. Experimental results show that V-SKP achieves a significant level of parameter and FLOPs reduction with acceptable accuracy degradation in various networks, including ResNet-50, and facilitates real acceleration effects on the GPUs, unlike conventional kernel pruning techniques.
BibTeX
@article{koo2023v,
  author = {Kwanghyun Koo and Hyun Kim},
  title = {{V-SKP: Vectorized Kernel-Based Structured Kernel Pruning for Accelerating Deep Convolutional Neural Networks}},
  journal = {IEEE Access},
  year = {2023},
  doi = {10.1109/access.2023.3326534},
}
Figure