AsymmNet: Towards Ultralight Convolution Neural Networks Using Asymmetrical Bottlenecks

Haojin Yang, Zhen Shen, Yucheng Zhao

Abstract

Deep convolutional neural networks (CNN) have achieved astonishing results in a large variety of applications. However, using these models on mobile or embedded devices is difficult due to the limited memory and computation resources. Recently, the inverted residual block has become the dominating solution for the architecture design of compact CNNs. In this work, we comprehensively investigate existing design concepts and rethink the functional characteristics of the two pointwise convolutions in inverted residuals. We propose a novel design, called asymmetrical bottlenecks. Precisely, we adjust the first pointwise convolution dimension, enrich the information flow by feature reuse, and migrate saved computations to the second pointwise convolution. Doing so, we can further improve accuracy without increasing computation overhead. The asymmetrical bottlenecks can be adopted as a drop-in replacement for existing CNN blocks. We can thus create AsymmNet by easily stacking those blocks according to proper depth and width conditions. Extensive experiments demonstrate that our proposed block design is more beneficial than the original inverted residual bottlenecks for mobile networks, especially for ultralight CNNs within the regime of <220M MAdds.

1. Introduction

The proliferation of deep learning in mobile and embedded applications has driven a significant research effort toward compact Convolutional Neural Network (CNN) architectures. While models like ResNet and VGG provide high accuracy, their computational demands are prohibitive for edge devices. The Inverted Residual Block (IRB), popularized by MobileNetV2, has emerged as the de facto standard for efficient design. By expanding the channel dimension, applying depthwise convolutions, and then projecting back, IRBs maintain a balance between performance and efficiency.

However, as we push into the "ultralight" regime—defined here as models with fewer than 220 million Multiply-Adds (MAdds)—the limitations of the standard IRB become apparent. Pointwise (1x1) convolutions, which handle channel-wise information exchange, consume the vast majority of the computational budget (up to 95%). In extremely constrained environments, the rigid symmetry of the standard bottleneck may not be the most efficient use of these limited resources.

Comparison of various compact CNN building blocks, highlighting the structural differences of the proposed asymmetrical bottleneck.

2. Methodology: The Asymmetrical Bottleneck

The core innovation of AsymmNet is the Asymmetrical Bottleneck Block. This design stems from a critical re-evaluation of the two pointwise (PW) convolutions within a standard inverted residual.

2.1 Computation Migration

In a standard IRB, the first pointwise convolution (PW1) expands the input to a higher-dimensional space, and the second (PW2) projects it back. The authors argue that PW2 is more critical for "expressiveness"—the ability of the network to learn complex features. By "pruning" or reducing the expansion ratio of PW1, they save computational cycles. These saved MAdds are then "migrated" to PW2, allowing it to handle a larger number of input channels and thereby improving the overall feature learning capacity of the block without increasing the total computational cost.

2.2 Feature Reuse and Information Flow

Simply pruning PW1 could lead to a loss of information. To mitigate this, AsymmNet employs a Feature Reuse mechanism. The original input features are concatenated directly with the output of the reduced PW1 layer. This "Concat" operation ensures that the subsequent depthwise convolution (DW) has access to a rich set of features, effectively bypassing the bottleneck for a portion of the information flow.

The evolution of the bottleneck design: (a) standard inverted residual, (b) the intermediate pruned block, and (c) the final asymmetrical block with feature reuse.

2.3 Architecture: AsymmNet-L and AsymmNet-S

The researchers developed two primary variants, AsymmNet-Large (L) and AsymmNet-Small (S). These were constructed by taking the established backbones of MobileNetV3 and replacing standard blocks with the new asymmetrical counterparts. This "drop-in" compatibility demonstrates the versatility of the design.

3. Experimental Evaluation

The effectiveness of AsymmNet was validated across several demanding computer vision tasks.

3.1 ImageNet Classification

On the ImageNet (ILSVRC 2012) benchmark, AsymmNet showed its greatest advantages at the smallest scales. At the 0.35x scale, AsymmNet-Small achieved a 55% Top-1 accuracy, a significant improvement over the 49.8% achieved by MobileNetV3-Small at a comparable computational cost.

3.2 Face Recognition and Other Tasks

The architecture's robustness was further tested on five face recognition datasets (LFW, CALFW, CPLFW, AgeDB-30, and VGGFace2). AsymmNet-S/L consistently outperformed MobileNetV3. Notably, the model achieved competitive results while being approximately 21 times smaller than MobileFaceNet. Similar performance gains were observed in action recognition (HMDB51), human pose estimation (COCO), and object detection (PASCAL VOC).

3.3 Real-world Latency

Beyond theoretical MAdds, the authors measured actual inference speed on an Android device equipped with a Snapdragon 855 processor. The "Pruned" version of the block achieved a latency of 2.9ms, outperforming MobileNetV3's 3.0ms at the smallest scale, proving that the theoretical efficiency translates to real-world performance.

4. Conclusion

AsymmNet demonstrates that rethinking the internal resource allocation of bottleneck blocks can yield significant performance gains for ultralight CNNs. By migrating computation to the feature-learning phase and utilizing smart feature reuse, the asymmetrical bottleneck provides a superior alternative to the standard inverted residual. Future work will explore the use of Neural Architecture Search (NAS) to further optimize the asymmetry rates and layer configurations across the network.

Summary updated

Abstract

Deep convolutional neural networks (CNN) have achieved astonishing results in a large variety of applications. However, using these models on mobile or embedded devices is difficult due to the limited memory and computation resources. Recently, the inverted residual block becomes the dominating solution for the architecture design of compact CNNs. In this work, we comprehensively investigated the existing design concepts, rethink the functional characteristics of two pointwise convolutions in the inverted residuals. We propose a novel design, called asymmetrical bottlenecks. Precisely, we adjust the first pointwise convolution dimension, enrich the information flow by feature reuse, and migrate saved computations to the second pointwise convolution. Doing so we can further improve the accuracy without increasing the computation overhead. The asymmetrical bottlenecks can be adopted as a drop-in replacement for the existing CNN blocks. We can thus create AsymmNet by easily stack those blocks according to proper depth and width conditions. Extensive experiments demonstrate that our proposed block design is more beneficial than the original inverted residual bottlenecks for mobile networks, especially useful for those ultralight CNNs within the regime of <220M MAdds. Code is available at: https://github.com/Spark001/AsymmNet
BibTeX
@inproceedings{yang2021asymmnet,
  author = {Haojin Yang and Zhen Shen and Yucheng Zhao},
  title = {{AsymmNet: Towards Ultralight Convolution Neural Networks Using Asymmetrical Bottlenecks}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=y9liYDzVwt},
}
Figure