CPWC: Contextual Point Wise Convolution for Object Recognition

Pratik Mazumder, Pravendra Singh, Vinay Namboodiri

Abstract

Convolutional layers are a major driving force behind the successes of deep learning. Pointwise convolution (PWC) is a 1x1 convolutional filter that is primarily used for parameter reduction. However, the PWC ignores the spatial information around the points it is processing. This design is by choice, in order to reduce the overall parameters and computations. However, we hypothesize that this shortcoming of PWC has a significant impact on the network performance. We propose an alternative design for pointwise convolution, which uses spatial information from the input efficiently. Our design significantly improves the performance of the networks without substantially increasing the number of parameters and computations. We experimentally show that our design results in significant improvement in the performance of the network for classification as well as detection.

Introduction

In modern deep learning, efficiency is paramount. To reduce the massive number of parameters and operations in architectures like ResNet or MobileNet, researchers rely heavily on Pointwise Convolutions (PWC)—filters with a $1 \times 1$ kernel. These are highly effective for channel-wise information mixing and bottlenecking. However, they possess a fundamental limitation: they are spatially "blind," processing each pixel in isolation without regard for its neighbors.

While the industry has largely accepted this loss of local context as a necessary trade-off for efficiency, this paper argues that spatial blindness limits network potential. The authors introduce Contextual Point Wise Convolution (CPWC), a lightweight module designed to re-introduce multi-scale spatial context into the $1 \times 1$ layer without the computational cost of standard $3 \times 3$ convolutions.

Methodology: Context Extraction

The core innovation of CPWC is a parallel "context extraction unit" that augments the standard $1 \times 1$ convolution rather than replacing it. The architecture consists of two primary stages designed to capture local and broader spatial information efficiently.

The CPWC architecture featuring parallel context extraction stages alongside a standard 1x1 convolution.
  1. Grouped Spatial Extraction (Stage 1): The input channels are divided into $Z$ groups. A $3 \times 3$ convolution is applied to each group independently. This grouped approach captures local spatial context while keeping the parameter count significantly lower than a full $3 \times 3$ convolution across all channels.
  2. Multi-Scale Expansion (Stage 2): The output of Stage 1 is processed by another $3 \times 3$ convolution. By stacking these layers, the network effectively achieves a $5 \times 5$ receptive field, capturing broader contextual cues.
  3. Integration: The outputs from the $1 \times 1$ convolution and the context extraction stages are combined via element-wise addition to produce the final output feature map.

Experimental Results

The authors evaluated CPWC across major benchmarks, including ImageNet for classification and MS COCO for object detection. The results consistently demonstrate that adding even a small amount of spatial context significantly boosts performance.

Performance comparison of CPWC against baseline models on the ImageNet dataset.

ImageNet Classification

As shown in the table above, a ResNet-50 equipped with CPWC achieves a Top-1 accuracy of 77.2%, matching the performance of the much deeper ResNet-101. Crucially, the CPWC version uses approximately 26.05M parameters, compared to the 44.55M parameters of ResNet-101—a reduction of over 40%. Similarly, for mobile-optimized architectures, CPWC improved MobileNetV2 (1.0x) accuracy from 72.0% to 73.0% with a negligible increase in FLOPs.

Object Detection and Ablation

Beyond classification, CPWC proved effective for object detection. Integrating CPWC into a Faster R-CNN backbone (ResNet-50) improved the Mean Average Precision (mAP) on MS COCO from 30.3 to 31.7.

Ablation studies further revealed that the original $1 \times 1$ convolution remains vital; removing it while keeping only the context unit leads to a significant performance drop. This confirms that CPWC acts as a powerful spatial complement to the channel-mixing capabilities of the pointwise layer.

Conclusion

CPWC demonstrates that spatial context is not an optional luxury for efficient networks. By using grouped convolutions to extract multi-scale information, CPWC allows models to achieve the performance of much deeper architectures with a fraction of the parameters. While future work may address hardware-specific optimizations for grouped operations, CPWC provides a clear path forward for making "blind" convolutions see the bigger picture.

Summary updated

Abstract

Convolutional layers are a major driving force behind the successes of deep learning. Pointwise convolution (PWC) is a 1x1 convolutional filter that is primarily used for parameter reduction. However, the PWC ignores the spatial information around the points it is processing. This design is by choice, in order to reduce the overall parameters and computations. However, we hypothesize that this shortcoming of PWC has a significant impact on the network performance. We propose an alternative design for pointwise convolution, which uses spatial information from the input efficiently. Our design significantly improves the performance of the networks without substantially increasing the number of parameters and computations. We experimentally show that our design results in significant improvement in the performance of the network for classification as well as detection.
BibTeX
@inproceedings{mazumder2019cpwc,
  author = {Pratik Mazumder and Pravendra Singh and Vinay Namboodiri},
  title = {{CPWC: Contextual Point Wise Convolution for Object Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  doi = {10.1109/icassp40776.2020.9054205},
}
Figure