Convolutional Neural Networks (CNNs) have achieved remarkable success in computer vision, yet they remain inherently sensitive to object scale. Recognizing an object when it fills the frame is significantly easier for a standard CNN than identifying the same object when it is small or partially obscured. Traditionally, researchers have addressed this through macro-architecture changes—designing complex networks with parallel branches (e.g., Inception), skip connections (e.g., FPN), or multiple resolutions (e.g., HRNet). While effective, these methods often increase parameter counts and computational overhead.
This paper introduces Poly-Scale Convolution (PSConv), a novel "drop-in" replacement for standard convolutional layers that addresses scale variance at the micro-architecture level. By incorporating multi-scale feature extraction directly into the kernel space, PSConv achieves the benefits of a feature pyramid without adding any extra parameters or theoretical computational complexity.
The PSConv Mechanism
The core innovation of PSConv is the mixing of a spectrum of different dilation rates within a single convolutional layer. In a standard convolution, every kernel in a layer shares the same receptive field. PSConv breaks this uniformity by varying dilation rates cyclically along the axis of input channels.
Key design principles include:
- Cyclic Dilation Rates: Dilation rates vary in a periodic pattern (e.g., 1, 2, 1, 4) across input channels. This ensures the layer samples the input at multiple scales simultaneously.
- Shift-based Strategy: To ensure that different output filters "see" different combinations of scales, the dilation pattern is shifted by one channel for each subsequent output filter.
The Kernel Lattice
This design creates a 2D "trellis" or lattice of dilation rates across the input and output channel dimensions. This structure allows the network to learn self-adaptive attention, dynamically routing features of different scales through the layer. Because it only changes the sampling locations of the weights (via dilation) and not the weights themselves, PSConv maintains the exact same parameter count as a vanilla convolution.
As shown in the comparison above, PSConv provides a much denser and more varied connectivity pattern than standard or group convolutions, allowing for a more granular approach to scale representation.
Experimental Results
The authors evaluated PSConv across several standard benchmarks, demonstrating its efficiency and effectiveness:
- ImageNet Classification: A PS-ResNet-50 achieved a ~1.7% improvement in top-1 error over the standard ResNet-50 baseline (21.1% vs 22.8%) with zero additional parameters. Remarkably, a PS-ResNet-50 performs comparably to a vanilla ResNet-101, effectively halving the required parameters for the same performance level.
- MS COCO Object Detection: When used as a backbone for Mask R-CNN, PSConv provided a significant boost. For a ResNet-50 backbone, the Average Precision (AP) increased from 37.3 to 39.4 (+2.1). Qualitative analysis showed that PSConv-based models are better at rejecting false alarms and more precise at bounding small-scale instances.
- CIFAR-100: The architecture demonstrated strong generalization ability on smaller, lower-resolution images.
Implementation and Efficiency
While theoretically identical in FLOPs to standard convolutions, PSConv faces practical challenges due to current deep learning libraries (like cuDNN) being optimized for uniform dilation. However, by using specialized implementations like Dilated Winograd Convolution (DWC), the authors reduced the inference overhead to just 1.06x compared to vanilla ResNet on a GPU. Future work involving customized CUDA kernels or specialized compilers like TVM could further bridge this gap.
Conclusion
PSConv demonstrates that granularity at the kernel level matters. By "squeezing" the functionality of a feature pyramid into a single compact layer, it offers a highly efficient path toward scale-robust computer vision. Its "plug-and-play" nature makes it compatible with other popular modules like Squeeze-and-Excitation (SE) blocks or Group Convolutions, providing a versatile tool for modern CNN design.
Summary updated