Abstract
Many effective solutions have been proposed to reduce the redundancy of models for inference acceleration. Nevertheless, common approaches mostly focus on eliminating less important filters or constructing efficient operations, while ignoring the pattern redundancy in feature maps. We reveal that many feature maps within a layer share similar but not identical patterns. However, it is difficult to identify if features with similar patterns are redundant or contain essential details. Therefore, instead of directly removing uncertain redundant features, we propose a split-based convolutional operation, namely SPConv, to tolerate features with similar patterns but require less computation. Specifically, we split input feature maps into the representative part and the uncertain redundant part, where intrinsic information is extracted from the representative part through relatively heavy computation while tiny hidden details in the uncertain redundant part are processed with some light-weight operation. To recalibrate and fuse these two groups of processed features, we propose a parameters-free feature fusion module. Moreover, our SPConv is formulated to replace the vanilla convolution in a plug-and-play way. Without any bells and whistles, experimental results on benchmarks demonstrate SPConv-equipped networks consistently outperform state-of-the-art baselines in both accuracy and inference time on GPU, with FLOPs and parameters dropped sharply.
Introduction: The Overlooked Redundancy
Despite the success of deep neural networks, their deployment is often hindered by massive parameter counts and high computational demands (FLOPs). Existing efficiency research typically focuses on pruning "unimportant" filters or using efficient architectures like Group-wise (GWC) or Depth-wise (DWC) convolutions. However, these methods often overlook pattern redundancy in feature maps, where many intermediate outputs share remarkably similar patterns.
Simply deleting these "redundant" maps is risky, as they may contain subtle, essential details. The challenge lies in exploiting this similarity to save computation without losing the "tiny hidden details" that contribute to high accuracy. SPConv addresses this by splitting channels and applying heterogeneous processing.
Methods: The SPConv Operation
The authors propose SPConv, a "plug-and-play" replacement for standard convolutions. It treats feature maps differently based on their complexity through a three-step process:
- Feature Split: Input channels ($C_{in}$) are split into a "representative" part (carrying core patterns) and a "redundant" part (carrying supplementary details) based on a ratio $\alpha$.
- Heterogeneous Processing:
- Representative Part: Processed using a parallel combination of Group-wise Convolution (GWC) and Point-wise Convolution (PWC). This ensures no information is lost due to the sparse nature of group convolutions.
- Redundant Part: Processed using a "cheap" $1 \times 1$ convolution to capture differences between similar maps at a fraction of the cost.
- Parameter-Free Feature Fusion: A module using Global Average Pooling (GAP) and a Softmax-based attention mechanism merges the two streams without adding learnable parameters, allowing the network to dynamically weight core patterns vs. fine details.
Experimental Results
The researchers evaluated SPConv across several standard benchmarks, including CIFAR-10, ImageNet-2012, and MS COCO.
ImageNet Classification
On the large-scale ImageNet-2012 dataset using ResNet-50, SPConv demonstrated superior efficiency. With a split ratio of $\alpha=1/2$, it reduced FLOPs and parameters by approximately 28%, while actually increasing accuracy by 0.35% over the baseline. Notably, SPConv-equipped models achieved faster inference times on GPUs (1.19ms) compared to both vanilla baselines (1.32ms) and other efficient variants like OctConv (3.51ms).
CIFAR-10 and Object Detection
- CIFAR-10: For VGG-16, SPConv achieved a massive 77% reduction in FLOPs while maintaining an accuracy of 93.94% (nearly identical to the 94.00% baseline).
- MS COCO: Replacing standard convolutions in Faster R-CNN and RetinaNet backbones led to slight improvements in Mean Average Precision (mAP) while significantly reducing the backbone's computational footprint. The authors noted a slight trade-off in detecting very large objects, likely due to replacing some $3 \times 3$ kernels with $1 \times 1$ kernels.
Conclusion
SPConv provides a hardware-friendly, efficient alternative to vanilla convolution. By intelligently splitting channels and applying heterogeneous processing, it achieves significant reductions in parameters and FLOPs while maintaining or improving performance across various computer vision tasks. Its "plug-and-play" nature makes it a practical tool for deploying high-performance models on resource-constrained devices.
Summary updated