Abstract
The Selective Convolutional Network (SCN) is a novel approach to object detection that slashes computational costs by teaching neural networks to ignore the background. By using a lightweight "Selective Module" to identify foreground regions, the network performs convolutions only where they matter, reducing calculations by up to 33% with almost no loss in accuracy.
Introduction
Modern object detectors, while accurate, are computationally expensive. Even efficient one-stage detectors like SSD perform convolution operations across the entire image—including vast, empty areas of sky, road, or wall that contain no objects of interest. This represents a significant waste of resources.
Human vision doesn't process every pixel of a scene with equal intensity; our brains use structures like the superior colliculus to quickly filter out irrelevant background and focus on significant targets. The Selective Convolutional Network (SCN) implements this "spatial pruning" in a deep learning framework that remains end-to-end trainable and easy to integrate into existing architectures.
Methods
The SCN architecture introduces two primary components to achieve efficiency: the Selective Module and Masked-Convolution.
Selective Module
The Selective Module is a "plug-and-play" branch attached to the shallow layers of a detector. It uses an encoder-decoder structure, incorporating dilated convolutions to maintain a wide field of view without a massive parameter increase. This module generates a binary saliency map that identifies potential foreground regions. The network can be trained using direct supervision (ground-truth bounding boxes) or indirect supervision (learning from the final detection loss).
Masked-Convolution
Standard convolutions are replaced by a masked version in the deeper layers of the network. As shown in the figure below, the process utilizes the im2col operation to transform image patches into matrix columns.
The saliency map acts as a filter, selecting only the spatial locations identified as foreground. Heavy matrix multiplication is performed exclusively on these selected columns. Finally, the results are scattered back into the full feature map using a col2im operation. This ensures that computational effort is concentrated solely on informative regions.
Key Insights
- Spatial vs. Channel Pruning: Unlike traditional model compression that removes network channels, SCN focuses on spatial pruning (width and height), which is more intuitive for visual tasks.
- Supervision Strategies: Indirect supervision (unsupervised mask learning) often yields higher accuracy because the network remains "conservative," avoiding the accidental exclusion of complex backgrounds where small objects might reside. Direct supervision is more aggressive in pruning, leading to higher compute savings but a slightly higher risk of missing targets.
- Zero-Cost Integration: Because the Selective Module shares features with the backbone, the overhead of generating the saliency map is negligible compared to the savings gained from masked convolutions.
Experimental Results
The researchers evaluated SCN using the PASCAL VOC and MS COCO benchmarks, applying it to the SSD (VGG-16) and Pelee detectors.
- PASCAL VOC: SCN reduced the GFLOPs of the Pelee detector by 33.1% and the SSD detector by approximately 20%. Accuracy (mAP) drop was minimal, often within 0.1% to 0.2%.
- MS COCO: On this more challenging dataset with many small objects, SCN achieved a 22.9% to 31.9% reduction in calculations, demonstrating robustness in cluttered scenes.
Conclusion
SCN effectively reduces the computational redundancy in object detection by focusing on foreground regions. While coarse bounding boxes can sometimes provide unreliable supervision at object edges, the overall framework provides a significant leap in efficiency. Future work may involve combining SCN with channel pruning and knowledge distillation to create even leaner, more powerful detection models.
Summary updated