Split to Be Slim: An Overlooked Redundancy in Vanilla Convolution

Qiulin Zhang, Zhuqing Jiang, Qishuo Lu, Jia'nan Han, Zhengxin Zeng, Shang-hua Gao, Aidong Men

Abstract

Many effective solutions have been proposed to reduce the redundancy of models for inference acceleration. Nevertheless, common approaches mostly focus on eliminating less important filters or constructing efficient operations, while ignoring the pattern redundancy in feature maps. We reveal that many feature maps within a layer share similar but not identical patterns. However, it is difficult to identify if features with similar patterns are redundant or contain essential details. Therefore, instead of directly removing uncertain redundant features, we propose a split based convolutional operation, namely SPConv, to tolerate features with similar patterns but require less computation. Specifically, we split input feature maps into the representative part and the uncertain redundant part, where intrinsic information is extracted from the representative part through relatively heavy computation while tiny hidden details in the uncertain redundant part are processed with some light-weight operation. To recalibrate and fuse these two groups of processed features, we propose a parameters-free feature fusion module. Moreover, our SPConv is formulated to replace the vanilla convolution in a plug-and-play way. Without any bells and whistles, experimental results on benchmarks demonstrate SPConv-equipped networks consistently outperform state-of-the-art baselines in both accuracy and inference time on GPU, with FLOPs and parameters dropped sharply.

Introduction

Deep Neural Networks (DNNs) are notoriously resource-hungry. While researchers have tried to slim them down using filter pruning or efficient operations (like Depthwise or Group-wise convolutions), an overlooked issue persists: pattern redundancy in feature maps. Visualization of intermediate outputs in standard architectures like ResNet reveals that many feature maps share strikingly similar patterns. Standard "vanilla" convolution treats every input channel as equally important, applying expensive $k \times k$ filters to all of them, which is computationally wasteful.

Previous attempts to address this often resulted in models that look efficient on paper (low FLOPs) but run slowly on actual GPUs due to fragmented memory access or complex operations. SPConv addresses this by tolerating redundancy rather than attempting to prune it, processing suspected redundant channels with significantly cheaper operations.

Methodology: The SPConv Module

The proposed SPConv (Split Convolution) divides the convolution process into three main steps to handle feature map redundancy efficiently.

Overview of the SPConv module architecture.
  1. Feature Splitting: Input channels ($C_{in}$) are split into two groups based on a ratio ($\alpha$): the Representative part ($\alpha C_{in}$) and the Redundant part ($(1-\alpha) C_{in}$).
  2. Asymmetric Processing:
    • Representative Branch: This branch extracts "intrinsic" information using a combination of Group-wise Convolution (GWC) and Point-wise Convolution (PWC). This captures spatial patterns while maintaining inter-channel connectivity.
  3. Redundant Branch: Since these maps are similar to the representative ones, they only need "tiny details." This branch uses a cheap $1 \times 1$ convolution (PWC) to process them.
  4. Parameter-Free Feature Fusion: To merge these two different views, a fusion module uses Global Average Pooling (GAP) and a Softmax-based attention mechanism to dynamically weigh and combine the features. This module adds zero extra parameters.

Because SPConv maintains the same input/output dimensions as a standard convolution, it is "plug-and-play"—it can be swapped into existing architectures like ResNet or VGG without changing the overall structure.

Experimental Results

The researchers evaluated SPConv across several standard benchmarks, including CIFAR-10, ImageNet-2012, and MS COCO.

Performance comparison on ImageNet-2012 using ResNet-50.

On ImageNet-2012 with ResNet-50, SPConv reduced parameters and FLOPs by approximately 28% while actually increasing Top-1 accuracy by 0.35%. Notably, SPConv-equipped models achieved faster real-world GPU inference than vanilla baselines (1.23ms vs 1.32ms), outperforming other efficient variants like GhostNet and OctConv in the trade-off between accuracy and throughput.

On CIFAR-10 with VGG-16, extreme efficiency settings ($\alpha=1/16$) reduced FLOPs and parameters by over 84% with less than a 0.6% drop in accuracy. For Object Detection on MS COCO, SPConv-equipped Faster R-CNN and RetinaNet models outperformed vanilla baselines in Mean Average Precision (mAP) while using significantly fewer resources.

Conclusion

SPConv offers a robust, plug-and-play alternative to vanilla convolution that effectively addresses pattern redundancy. By splitting features and applying asymmetric processing, it achieves significant reductions in computational cost while maintaining or improving accuracy. Future work may explore automated tuning of the split ratio $\alpha$ across different network stages and addressing the slight performance dip observed in detecting very large objects, likely due to the reduced receptive field of $1 \times 1$ kernels in the redundant branch.

Summary updated

Abstract

Many effective solutions have been proposed to reduce the redundancy of models for inference acceleration. Nevertheless, common approaches mostly focus on eliminating less important filters or constructing efficient operations, while ignoring the pattern redundancy in feature maps. We reveal that many feature maps within a layer share similar but not identical patterns. However, it is difficult to identify if features with similar patterns are redundant or contain essential details. Therefore, instead of directly removing uncertain redundant features, we propose a split based convolutional operation, namely SPConv, to tolerate features with similar patterns but require less computation. Specifically, we split input feature maps into the representative part and the uncertain redundant part, where intrinsic information is extracted from the representative part through relatively heavy computation while tiny hidden details in the uncertain redundant part are processed with some light-weight operation. To recalibrate and fuse these two groups of processed features, we propose a parameters-free feature fusion module. Moreover, our SPConv is formulated to replace the vanilla convolution in a plug-and-play way. Without any bells and whistles, experimental results on benchmarks demonstrate SPConv-equipped networks consistently outperform state-of-the-art baselines in both accuracy and inference time on GPU, with FLOPs and parameters dropped sharply.
BibTeX
@inproceedings{zhang2020split,
  author = {Qiulin Zhang and Zhuqing Jiang and Qishuo Lu and Jia'nan Han and Zhengxin Zeng and Shang-hua Gao and Aidong Men},
  title = {{Split to Be Slim: An Overlooked Redundancy in Vanilla Convolution}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=2wsuehTXVV},
}
Figure