PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer

Duo Li, Anbang Yao, Qifeng Chen

Convolutional Neural Networks (CNNs) have achieved remarkable success in computer vision, yet they remain inherently sensitive to object scale. Recognizing an object when it fills the frame is significantly easier for a standard CNN than identifying the same object when it is small or partially obscured. Traditionally, researchers have addressed this through macro-architecture changes—designing complex networks with parallel branches (e.g., Inception), skip connections (e.g., FPN), or multiple resolutions (e.g., HRNet). While effective, these methods often increase parameter counts and computational overhead.

This paper introduces Poly-Scale Convolution (PSConv), a novel "drop-in" replacement for standard convolutional layers that addresses scale variance at the micro-architecture level. By incorporating multi-scale feature extraction directly into the kernel space, PSConv achieves the benefits of a feature pyramid without adding any extra parameters or theoretical computational complexity.

The PSConv Mechanism

The core innovation of PSConv is the mixing of a spectrum of different dilation rates within a single convolutional layer. In a standard convolution, every kernel in a layer shares the same receptive field. PSConv breaks this uniformity by varying dilation rates cyclically along the axis of input channels.

3D visualization of PSConv showing varying dilation rates across kernel channels.

Key design principles include:

  • Cyclic Dilation Rates: Dilation rates vary in a periodic pattern (e.g., 1, 2, 1, 4) across input channels. This ensures the layer samples the input at multiple scales simultaneously.
  • Shift-based Strategy: To ensure that different output filters "see" different combinations of scales, the dilation pattern is shifted by one channel for each subsequent output filter.

The Kernel Lattice

This design creates a 2D "trellis" or lattice of dilation rates across the input and output channel dimensions. This structure allows the network to learn self-adaptive attention, dynamically routing features of different scales through the layer. Because it only changes the sampling locations of the weights (via dilation) and not the weights themselves, PSConv maintains the exact same parameter count as a vanilla convolution.

Comparison of connectivity patterns between standard, group, and poly-scale convolutions.

As shown in the comparison above, PSConv provides a much denser and more varied connectivity pattern than standard or group convolutions, allowing for a more granular approach to scale representation.

Experimental Results

The authors evaluated PSConv across several standard benchmarks, demonstrating its efficiency and effectiveness:

  • ImageNet Classification: A PS-ResNet-50 achieved a ~1.7% improvement in top-1 error over the standard ResNet-50 baseline (21.1% vs 22.8%) with zero additional parameters. Remarkably, a PS-ResNet-50 performs comparably to a vanilla ResNet-101, effectively halving the required parameters for the same performance level.
  • MS COCO Object Detection: When used as a backbone for Mask R-CNN, PSConv provided a significant boost. For a ResNet-50 backbone, the Average Precision (AP) increased from 37.3 to 39.4 (+2.1). Qualitative analysis showed that PSConv-based models are better at rejecting false alarms and more precise at bounding small-scale instances.
  • CIFAR-100: The architecture demonstrated strong generalization ability on smaller, lower-resolution images.

Implementation and Efficiency

While theoretically identical in FLOPs to standard convolutions, PSConv faces practical challenges due to current deep learning libraries (like cuDNN) being optimized for uniform dilation. However, by using specialized implementations like Dilated Winograd Convolution (DWC), the authors reduced the inference overhead to just 1.06x compared to vanilla ResNet on a GPU. Future work involving customized CUDA kernels or specialized compilers like TVM could further bridge this gap.

Conclusion

PSConv demonstrates that granularity at the kernel level matters. By "squeezing" the functionality of a feature pyramid into a single compact layer, it offers a highly efficient path toward scale-robust computer vision. Its "plug-and-play" nature makes it compatible with other popular modules like Squeeze-and-Excitation (SE) blocks or Group Convolutions, providing a versatile tool for modern CNN design.

Summary updated

BibTeX
@inproceedings{li2020psconv,
  author = {Duo Li and Anbang Yao and Qifeng Chen},
  title = {{PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=yTOJ95Mhziv},
}
Figure