PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer

Duo Li, Anbang Yao, Qifeng Chen

This academic summary explores the paper "PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer," which introduces a novel way to make Convolutional Neural Networks (CNNs) more robust to changes in object scale without increasing model size.

TL;DR

PSConv is a "drop-in" replacement for standard convolutional layers that incorporates multi-scale feature extraction directly into the kernel space. By cyclically varying dilation rates across individual kernels within a single layer, it mimics the behavior of a complex feature pyramid without adding any extra parameters or theoretical computational complexity.

What Problem Does This Paper Address?

Convolutional Neural Networks are notoriously "scale-sensitive." An object might be recognized easily when it fills the frame, but the same network might struggle if the object is tiny or partially obscured.

To fix this, researchers usually look at macro-architecture—designing complex networks with parallel branches (like Inception), skip connections (like FPN), or multiple resolutions (like HRNet). While effective, these methods often require significant manual tuning, increase the number of parameters, or add computational overhead. The authors of PSConv argue that we have overlooked the micro-architecture: the granular space of individual kernels. They aim to solve scale variance within a single, compact layer.

Methods

The core innovation is Poly-Scale Convolution (PSConv). In a standard convolution, every kernel in a layer has the same receptive field. In PSConv, the authors mix a spectrum of different dilation rates within the same layer.

Key design principles include:

  • Cyclic Dilation Rates: Along the axis of input channels, dilation rates vary in a periodic pattern (e.g., 1, 2, 1, 4). This ensures that the layer samples the input at multiple scales simultaneously.
  • Shift-based Strategy: To ensure that different output filters "see" different combinations of scales, the dilation pattern is shifted by one channel for each subsequent output filter.
  • The Kernel Lattice: This creates a 2D "trellis" of dilation rates across the input and output channel dimensions. This structure allows the network to learn self-adaptive attention, dynamically routing features of different scales through the layer.

Because it only changes the sampling locations of the weights (via dilation) and not the weights themselves, PSConv uses the exact same number of parameters as a vanilla convolution.

Datasets

The authors evaluated PSConv across several standard benchmarks:

  • ImageNet (ILSVRC 2012): Used for large-scale image classification.
  • MS COCO 2017: Used for object detection and instance segmentation (tasks where scale variance is a major challenge).
  • CIFAR-100: Used to test the generalization ability of the architecture on smaller, lower-resolution images.

Key Insights and Ideas

  • Granularity Matters: You don't need a massive, multi-branch network to handle different scales. You can achieve similar effects by simply being smarter about how individual kernels sample pixels.
  • Efficiency through Condensation: A single PSConv layer can effectively "squeeze" the functionality of a feature pyramid. For example, a PS-ResNet-50 (50 layers) achieves performance comparable to a vanilla ResNet-101, effectively halving the required parameters.
  • Dynamic Routing: By spreading scale information across the "kernel lattice," the network can better handle non-rigid shapes or scenes containing multiple objects of vastly different sizes.
  • Orthogonality: PSConv is "plug-and-play." It can be combined with other popular modules like Squeeze-and-Excitation (SE) blocks or Group Convolutions without conflict.

Key Results

  • ImageNet Classification: PS-ResNet-50 showed a ~1.7% improvement in top-1 error over the standard ResNet-50 baseline (21.1% vs 22.8%) with zero additional parameters.
  • COCO Object Detection: When used as a backbone for Mask R-CNN, PSConv provided a significant boost. For a ResNet-50 backbone, the Average Precision (AP) increased from 37.3 to 39.4 (+2.1).
  • Qualitative Success: Visualization of detection results showed that PSConv-based models are much better at rejecting "false alarms" (like mistaking a refrigerator for a large-scale object) and are more precise at bounding small-scale instances.
  • Inference Speed: While theoretically identical in FLOPs, standard deep learning libraries (like cuDNN) are not perfectly optimized for mixed dilation. However, using specialized implementations like Dilated Winograd Convolution (DWC), the authors reduced the inference overhead to just 1.06x compared to vanilla ResNet on a GPU.

Limitations and Future Work

  • Hardware Optimization: The primary limitation is that current GPU kernels are optimized for uniform dilation. To see the full speed benefits of PSConv, customized CUDA kernels or specialized compilers (like TVM) are required.
  • Dilation Limits: The authors found that increasing dilation rates too aggressively (e.g., using a rate of 8) can actually hurt performance by introducing "noise" from irrelevant pixels. Finding the optimal "spectrum" of rates remains a heuristic process.
  • Learnable Patterns: While the authors experimented with a learnable binary mask for dilation, the paper primarily focuses on fixed cyclic patterns. Future work could involve fully differentiable scale-selection mechanisms within the kernel.

Summary updated

Abstract

Despite their strong modeling capacities, Convolutional Neural Networks (CNNs) are often scale-sensitive. For enhancing the robustness of CNNs to scale variance, multi-scale feature fusion from different layers or filters attracts great attention among existing solutions, while the more granular kernel space is overlooked. We bridge this regret by exploiting multi-scale features in a finer granularity. The proposed convolution operation, named Poly-Scale Convolution (PSConv), mixes up a spectrum of dilation rates and tactfully allocates them in the individual convolutional kernels of each filter regarding a single convolutional layer. Specifically, dilation rates vary cyclically along the axes of input and output channels of the filters, aggregating features over a wide range of scales in a neat style. PSConv could be a drop-in replacement of the vanilla convolution in many prevailing CNN backbones, allowing better representation learning without introducing additional parameters and computational complexities. Comprehensive experiments on the ImageNet and MS COCO benchmarks validate the superior performance of PSConv. Code and models are available at https://github.com/d-li14/PSConv .
BibTeX
@inproceedings{li2020psconv,
  author = {Duo Li and Anbang Yao and Qifeng Chen},
  title = {{PSConv: Squeezing Feature Pyramid into One Compact Poly-Scale Convolutional Layer}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=yTOJ95Mhziv},
}

Loading related content...

Figure