Efficient Fusion of Sparse and Complementary Convolutions for Object Recognition and Detection

Chun-Fu Chen, Quanfu Fan, Marco Pistoia, Gwo Giun LeearXiv 2018 · arXiv

Abstract

Deep learning models are increasingly deployed on resource-constrained devices, necessitating efficient model compression techniques. This paper presents scFusion, a novel approach that utilizes hand-crafted, spatially complementary (SC) sparse kernels to create compact Convolutional Neural Networks (CNNs). By fusing these structured sparse patterns in parallel, the proposed scFusion module achieves up to a 12.4× reduction in model size and a 3× speedup for object detection tasks with minimal impact on accuracy.

Introduction

The proliferation of deep learning in real-time IoT systems and mobile devices has driven the demand for smaller, faster CNNs. While traditional model compression techniques like pruning can significantly reduce parameter counts, they often result in irregular sparsity patterns. These "random" gaps are difficult for standard hardware like GPUs to exploit efficiently, as they are optimized for dense computations.

Problem Statement: The Challenge of Irregularity

Most existing compression methods face a practical hurdle: hardware-software mismatch. Irregularly pruned weights require specialized engineering to achieve actual wall-clock speedups. To address this, the authors propose moving away from learned random sparsity toward structured, hand-crafted sparsity that is inherently hardware-friendly.

Methodology: The scFusion Module

The core innovation is the scFusion module, which leverages Spatial Complementary (SC) kernels to maintain full receptive field coverage while reducing computation.

1. Checkerboard Kernels

Instead of standard dense kernels, scFusion employs pairs of sparse kernels, $W_{even}$ and $W_{odd}$, arranged in a checkerboard pattern. $W_{even}$ contains weights only on even-indexed positions, while $W_{odd}$ occupies the odd positions.

Comparison between regular kernels and spatially complementary (SC) kernels.

2. Parallel Kernel Fusion

Unlike sequential stacking of sparse kernels, which can lead to "blind spots" in the receptive field, scFusion processes these kernels in parallel. Their outputs are added together, ensuring that every pixel in the receptive field contributes to the final feature map.

3. Inverse and Channel-wise Fusion

To enrich feature representation without significantly increasing parameters, an "inverse" operation is applied to generate symmetric feature maps. Finally, a $1 \times 1$ pointwise convolution linearly combines these sparse and joint features into a robust final representation.

The architecture of the scFusion module, illustrating parallel sparse convolutions and fusion steps.

Experimental Results

The scFusion approach was validated on benchmarks including CIFAR-10, ImageNet, PASCAL VOC, and MS COCO across various architectures like AlexNet, VGG-16, and ResNet.

  • Model Efficiency: On ImageNet, a VGG-16 model enhanced with scFusion-8 and Deep Compression was 13.45× smaller than the baseline.
  • Object Detection Performance: In Faster R-CNN, the scFusion variant achieved a 3× speedup and 12.4× size reduction compared to a VGG-16 baseline, maintaining a competitive 73.4 mAP.
  • Hardware Acceleration: The deterministic nature of SC kernels allowed for a ~2× real-world speedup on an Nvidia K80 GPU, demonstrating its practical utility.

Conclusion

scFusion offers a compelling balance between efficiency and performance by utilizing structured sparsity. By ensuring full receptive field coverage through parallel fusion and leveraging hardware-friendly patterns, it provides a scalable solution for deploying high-performance CNNs on edge devices. Future work may explore combining scFusion with advanced quantization and distillation techniques for even greater compression.

Summary updated

Abstract

We propose a new method to create compact convolutional neural networks (CNNs) by exploiting sparse convolutions. Different from previous works that learn sparsity in models, we directly employ hand-crafted kernels with regular sparse patterns, which result in the computational gain in practice without sophisticated and dedicated software or hardware. The core of our approach is an efficient network module that linearly combines sparse kernels to yield feature representations as strong as those from regular kernels. We integrate this module into various network architectures and demonstrate its effectiveness on three vision tasks, object classification, localization and detection. For object classification and localization, our approach achieves comparable or better performance than several baselines and related works while providing lower computational costs with fewer parameters (on average, a $2-4\times$ reduction of convolutional parameters and computation). For object detection, our approach leads to a VGG-16-based Faster RCNN detector that is 12.4$\times$ smaller and about 3$\times$ faster than the baseline.
BibTeX
@misc{chen2018efficient,
  author = {Chun-Fu Chen and Quanfu Fan and Marco Pistoia and Gwo Giun Lee},
  title = {{Efficient Fusion of Sparse and Complementary Convolutions for Object Recognition and Detection}},
  year = {2018},
  eprint = {1808.02167},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1808.02167},
}
Figure