DiCENet: Dimension-wise Convolutions for Efficient Networks

Sachin Mehta, Hannaneh Hajishirzi, Mohammad RastegariIEEE TPAMI 2019 · DOI · OpenReview

Abstract

We introduce a novel and generic convolutional unit, DiCE unit, that is built using dimension-wise convolutions and dimension-wise fusion. The dimension-wise convolutions apply light-weight convolutional filtering across each dimension of the input tensor while dimension-wise fusion efficiently combines these dimension-wise representations; allowing the DiCE unit to efficiently encode spatial and channel-wise information contained in the input tensor. The DiCE unit is simple and can be seamlessly integrated with any architecture to improve its efficiency and performance. Compared to depth-wise separable convolutions, the DiCE unit shows significant improvements across different architectures. When DiCE units are stacked to build the DiCENet model, we observe significant improvements over state-of-the-art models across various computer vision tasks including image classification, object detection, and semantic segmentation. On the ImageNet dataset, the DiCENet delivers 2-4 percent higher accuracy than state-of-the-art manually designed models (e.g., MobileNetv2 and ShuffleNetv2). Also, DiCENet generalizes better to tasks (e.g., object detection) that are often used in resource-constrained devices in comparison to state-of-the-art separable convolution-based efficient networks, including neural search-based methods (e.g., MobileNetv3 and MixNet).

Introduction

In the quest for efficient deep learning, depth-wise separable convolutions (DWSC) have become the gold standard for mobile-friendly architectures. DWSC splits a standard convolution into two steps: a spatial (depth-wise) filter and a point-wise (1x1) convolution to mix channels. However, a hidden bottleneck exists: in architectures like ShuffleNetv2 and MobileNetv2, these 1x1 layers account for nearly 90% of the total operations. While necessary to encode channel-wise relationships, they are computationally heavy. DiCENet addresses this by asking: Can we encode both spatial and channel information more efficiently by looking at the input tensor from every possible dimension?

The DiCE Unit: Dimension-Wise Convolution and Fusion

The core innovation is the DiCE (Dimension-wise Convolution and Fusion) unit, which replaces traditional convolutions with dimension-wise filtering and fusion.

The DiCE Unit architecture, detailing the DimConv stage (filtering across depth, width, and height) and the DimFuse stage (local and global fusion).

The unit consists of two main components:

  1. DimConv (Dimension-wise Convolution): Instead of only applying filters across the channel dimension (depth-wise), DimConv applies light-weight filters across all dimensions independently: depth-wise ($K_D$), width-wise ($K_W$), and height-wise ($K_H$). This allows the network to capture local patterns from three different perspectives.
  2. DimFuse (Dimension-wise Fusion): To combine these representations without the heavy cost of a standard 1x1 convolution, DiCE uses a two-stage fusion. Local Fusion employs a grouped point-wise convolution to mix information within local groups, while Global Fusion uses global pooling and fully connected layers to scale features based on global context, similar to Squeeze-and-Excitation blocks.

To ensure practical efficiency, the authors developed a custom CUDA kernel that processes all dimensions in a single pass, maximizing GPU thread utilization compared to sequential kernel calls.

Experimental Evaluation

DiCENet was evaluated across ImageNet-1K for classification, MS-COCO for detection, and PASCAL VOC for segmentation.

Performance comparison showing DiCENet outperforming both manually designed and NAS-based models in both ImageNet accuracy and MS COCO object detection precision.

On ImageNet, DiCENet achieved 75.7% top-1 accuracy with ~300 MFLOPs, outperforming MobileNetv2 and ShuffleNetv2 by approximately 4%. More importantly, DiCENet demonstrates superior generalization to dense tasks. When used as a backbone for SSD on MS-COCO, it reached 25.1 mAP, significantly higher than neural architecture search (NAS) based models like MobileNetv3 (22.0 mAP) and MixNet (22.3 mAP).

Discussion and Future Work

The success of DiCENet highlights that the 1x1 point-wise bottleneck is a significant area for optimization. By replacing it with DimFuse, the computational cost of fusion is reduced by roughly 3x. Furthermore, the manual design of the DiCE unit proves that fundamental improvements in convolutional building blocks can outperform or match complex NAS-derived architectures.

Current limitations include a reliance on custom CUDA kernels for peak efficiency, which may not yet be optimized for mobile CPUs or specialized NPUs. Future work could involve integrating the DiCE unit into NAS search spaces to discover even more optimized configurations or developing specialized implementations for edge hardware.

Summary updated

Abstract

We introduce a novel and generic convolutional unit, DiCE unit, that is built using dimension-wise convolutions and dimension-wise fusion. The dimension-wise convolutions apply light-weight convolutional filtering across each dimension of the input tensor while dimension-wise fusion efficiently combines these dimension-wise representations; allowing the DiCE unit to efficiently encode spatial and channel-wise information contained in the input tensor. The DiCE unit is simple and can be seamlessly integrated with any architecture to improve its efficiency and performance. Compared to depth-wise separable convolutions, the DiCE unit shows significant improvements across different architectures. When DiCE units are stacked to build the DiCENet model, we observe significant improvements over state-of-the-art models across various computer vision tasks including image classification, object detection, and semantic segmentation. On the ImageNet dataset, the DiCENet delivers 2-4% higher accuracy than state-of-the-art manually designed models (e.g., MobileNetv2 and ShuffleNetv2). Also, DiCENet generalizes better to tasks (e.g., object detection) that are often used in resource-constrained devices in comparison to state-of-the-art separable convolution-based efficient networks, including neural search-based methods (e.g., MobileNetv3 and MixNet. Our source code in PyTorch is open-source and is available at https://github.com/sacmehta/EdgeNets/
BibTeX
@article{mehta2019dicenet,
  author = {Sachin Mehta and Hannaneh Hajishirzi and Mohammad Rastegari},
  title = {{DiCENet: Dimension-wise Convolutions for Efficient Networks}},
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
  year = {2019},
  doi = {10.1109/tpami.2020.3041871},
}
Figure