Abstract
In many real-time applications, the deployment of deep neural networks is constrained by high computational cost, making efficient lightweight neural networks a critical area of research. In this paper, we propose using depthwise convolution (DWC) to expand the number of channels in a bottleneck block, a method that proves more efficient than standard 1×1 pointwise convolution (PWC). We introduce the Pointwise-Standard-Depthwise network (PSDNet), which utilizes DWC for channel expansion to achieve fewer parameters, lower computational cost, and higher accuracy than ResNet on CIFAR datasets. Furthermore, we propose the Depthwise-Pointwise-Depthwise inverted bottleneck block (DPD block) and the corresponding DPDNet. DPDNet requires only about 60% of the parameters of MobileNetV2 for networks of the same depth while achieving comparable accuracy. Our findings indicate that networks emphasizing spatial information extraction via DWC outperform those focused on channel combination via PWC, suggesting a more effective allocation of parameter budgets for lightweight models.
Introduction
Deep learning models, while powerful, often face a "weight" problem that hinders their deployment on edge devices with limited computational resources. Standard architectures like ResNet and MobileNetV2 utilize "bottleneck" blocks to manage complexity. These blocks typically use 1×1 pointwise convolutions (PWC) to reduce dimensionality (squeeze) and then expand it back. However, as networks grow wider, these 1×1 expansion layers become significant sources of parameter redundancy. This research addresses this inefficiency by leveraging depthwise convolutions (DWC)—which are computationally leaner—to handle channel expansion.
Architectural Innovations
The core contribution of this work is the shift from PWC-based expansion to DWC-based expansion. The mathematical justification lies in the parameter ratio: a 1×1 expansion layer uses approximately $C$ times more parameters than a 3×3 DWC expansion layer for a given channel count $C$, specifically a ratio of roughly $C/9$. Since $C$ is typically much larger than 9, DWC expansion is inherently more efficient.
PSDNet (Pointwise-Standard-Depthwise)
PSDNet evolves the classic ResNet bottleneck. In a standard ResNet block, the sequence is 1×1 squeeze, 3×3 standard convolution, and 1×1 expansion. PSDNet replaces the final 1×1 expansion with a 3×3 Depthwise Convolution, significantly reducing the parameter footprint while maintaining representational power.
DPDNet (Depthwise-Pointwise-Depthwise)
Inspired by the inverted bottleneck of MobileNetV2, the DPD block utilizes a DWC for initial channel expansion, followed by a PWC to combine and squeeze channels, and a final DWC for spatial filtering. This structure prioritizes spatial feature extraction over channel-wise combination.
Experimental Evaluation
The proposed architectures were evaluated on CIFAR-10, CIFAR-100, CINIC-10, and ImageNet datasets.
- PSDNet50 vs. ResNet50: On CIFAR-10, PSDNet50 utilized 20% fewer parameters (1.6M vs. 2.0M) and 34% fewer FLOPs than ResNet50, yet achieved a higher accuracy of 93.87% compared to 92.95%.
- DPDNet vs. MobileNetV2: DPDNet achieved nearly identical accuracy to MobileNetV2 on CIFAR datasets while being significantly leaner. On ImageNet, DPDNet reached a 72.0% Top-1 accuracy, matching MobileNetV2's performance with a much more efficient profile.
- Flexibility: DPDNet introduces two hyperparameters—$\alpha$ (width multiplier) and $m$ (channel expansion factor)—allowing for precise trade-offs between accuracy and computational cost for diverse hardware constraints.
Key Insights and Conclusion
A primary finding of this study is that extracting spatial information is more critical than combining channel information for lightweight models. Networks with more DWC layers consistently outperformed those with more PWC layers. By rethinking the expansion mechanism in bottleneck blocks, PSDNet and DPDNet provide a new blueprint for designing efficient, high-performance neural networks for real-time applications. Future work will involve scaling these architectures to even deeper structures and evaluating their performance in downstream tasks like object detection and semantic segmentation.
Summary updated