PSDNet and DPDNet: Efficient channel expansion, Depthwise-Pointwise-Depthwise Inverted Bottleneck Block

Guoqing Li, Meng Zhang, Qianru Zhang, Ziyang Chen, Wen‐Zhao Liu, Jiaojie Li, Xuzhao Shen, Jianjun Li, Zhenyu Zhu, Chau Yuen

Abstract

In many real-time applications, the deployment of deep neural networks is constrained by high computational cost, making efficient lightweight neural networks a critical area of research. In this paper, we propose using depthwise convolution (DWC) to expand the number of channels in a bottleneck block, a method that proves more efficient than standard 1×1 pointwise convolution (PWC). We introduce the Pointwise-Standard-Depthwise network (PSDNet), which utilizes DWC for channel expansion to achieve fewer parameters, lower computational cost, and higher accuracy than ResNet on CIFAR datasets. Furthermore, we propose the Depthwise-Pointwise-Depthwise inverted bottleneck block (DPD block) and the corresponding DPDNet. DPDNet requires only about 60% of the parameters of MobileNetV2 for networks of the same depth while achieving comparable accuracy. Our findings indicate that networks emphasizing spatial information extraction via DWC outperform those focused on channel combination via PWC, suggesting a more effective allocation of parameter budgets for lightweight models.

Introduction

Deep learning models, while powerful, often face a "weight" problem that hinders their deployment on edge devices with limited computational resources. Standard architectures like ResNet and MobileNetV2 utilize "bottleneck" blocks to manage complexity. These blocks typically use 1×1 pointwise convolutions (PWC) to reduce dimensionality (squeeze) and then expand it back. However, as networks grow wider, these 1×1 expansion layers become significant sources of parameter redundancy. This research addresses this inefficiency by leveraging depthwise convolutions (DWC)—which are computationally leaner—to handle channel expansion.

Architectural Innovations

The core contribution of this work is the shift from PWC-based expansion to DWC-based expansion. The mathematical justification lies in the parameter ratio: a 1×1 expansion layer uses approximately $C$ times more parameters than a 3×3 DWC expansion layer for a given channel count $C$, specifically a ratio of roughly $C/9$. Since $C$ is typically much larger than 9, DWC expansion is inherently more efficient.

PSDNet (Pointwise-Standard-Depthwise)

PSDNet evolves the classic ResNet bottleneck. In a standard ResNet block, the sequence is 1×1 squeeze, 3×3 standard convolution, and 1×1 expansion. PSDNet replaces the final 1×1 expansion with a 3×3 Depthwise Convolution, significantly reducing the parameter footprint while maintaining representational power.

Structural comparison between the standard ResNet bottleneck and the proposed PSDNet block.

DPDNet (Depthwise-Pointwise-Depthwise)

Inspired by the inverted bottleneck of MobileNetV2, the DPD block utilizes a DWC for initial channel expansion, followed by a PWC to combine and squeeze channels, and a final DWC for spatial filtering. This structure prioritizes spatial feature extraction over channel-wise combination.

Comparison of the inverted bottleneck in MobileNetV2 versus the proposed DPD block in DPDNet.

Experimental Evaluation

The proposed architectures were evaluated on CIFAR-10, CIFAR-100, CINIC-10, and ImageNet datasets.

  • PSDNet50 vs. ResNet50: On CIFAR-10, PSDNet50 utilized 20% fewer parameters (1.6M vs. 2.0M) and 34% fewer FLOPs than ResNet50, yet achieved a higher accuracy of 93.87% compared to 92.95%.
  • DPDNet vs. MobileNetV2: DPDNet achieved nearly identical accuracy to MobileNetV2 on CIFAR datasets while being significantly leaner. On ImageNet, DPDNet reached a 72.0% Top-1 accuracy, matching MobileNetV2's performance with a much more efficient profile.
  • Flexibility: DPDNet introduces two hyperparameters—$\alpha$ (width multiplier) and $m$ (channel expansion factor)—allowing for precise trade-offs between accuracy and computational cost for diverse hardware constraints.

Key Insights and Conclusion

A primary finding of this study is that extracting spatial information is more critical than combining channel information for lightweight models. Networks with more DWC layers consistently outperformed those with more PWC layers. By rethinking the expansion mechanism in bottleneck blocks, PSDNet and DPDNet provide a new blueprint for designing efficient, high-performance neural networks for real-time applications. Future work will involve scaling these architectures to even deeper structures and evaluating their performance in downstream tasks like object detection and semantic segmentation.

Summary updated

Abstract

In many real-time applications, the deployment of deep neural networks is constrained by high computational cost and efficient lightweight neural networks are widely concerned. In this paper, we propose that depthwise convolution (DWC) is used to expand the number of channels in a bottleneck block, which is more efficient than 1 x 1 convolution. The proposed Pointwise-Standard-Depthwise network (PSDNet) based on channel expansion with DWC has fewer number of parameters, less computational cost and higher accuracy than corresponding ResNet on CIFAR datasets. To design more efficient lightweight concolutional neural netwok, Depthwise-Pointwise-Depthwise inverted bottleneck block (DPD block) is proposed and DPDNet is designed by stacking DPD block. Meanwhile, the number of parameters of DPDNet is only about 60% of that of MobileNetV2 for networks with the same number of layers, but can achieve approximated accuracy. Additionally, two hyperparameters of DPDNet can make the trade-off between accuracy and computational cost, which makes DPDNet suitable for diverse tasks. Furthermore, we find the networks with more DWC layers outperform the networks with more 1x1 convolution layers, which indicates that extracting spatial information is more important than combining channel information.
BibTeX
@misc{li2019psdnet,
  author = {Guoqing Li and Meng Zhang and Qianru Zhang and Ziyang Chen and Wen‐Zhao Liu and Jiaojie Li and Xuzhao Shen and Jianjun Li and Zhenyu Zhu and Chau Yuen},
  title = {{PSDNet and DPDNet: Efficient channel expansion, Depthwise-Pointwise-Depthwise Inverted Bottleneck Block}},
  year = {2019},
  eprint = {1909.01026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1909.01026},
}
Figure