Abstract
Deep convolutional neural networks (CNN) have achieved astonishing results in a large variety of applications. However, using these models on mobile or embedded devices is difficult due to the limited memory and computation resources. Recently, the inverted residual block becomes the dominating solution for the architecture design of compact CNNs. In this work, we comprehensively investigated the existing design concepts, rethink the functional characteristics of two pointwise convolutions in the inverted residuals. We propose a novel design, called asymmetrical bottlenecks. Precisely, we adjust the first pointwise convolution dimension, enrich the information flow by feature reuse, and migrate saved computations to the second pointwise convolution. By doing so we can further improve the accuracy without increasing the computation overhead. The asymmetrical bottlenecks can be adopted as a drop-in replacement for the existing CNN blocks. We can thus create AsymmNet by easily stack those blocks according to proper depth and width conditions. Extensive experiments demonstrate that our proposed block design is more beneficial than the original inverted residual bottlenecks for mobile networks, especially useful for those ultralight CNNs within the regime of <220M MAdds. Code is available at https://github.com/Spark001/AsymmNet
Introduction: The Challenge of Edge AI
While Deep Convolutional Neural Networks (CNNs) have revolutionized computer vision, their massive appetite for memory and computation makes them difficult to deploy on resource-constrained "edge" devices like smartphones and IoT sensors.
The current industry standard for compact CNNs is the inverted residual block (popularized by MobileNetV2). This block typically follows a "sandwich" structure: a pointwise (1x1) convolution expands the channel count, a depthwise convolution handles spatial features, and a final pointwise convolution projects the data back down.
The authors of AsymmNet argue that this standard design treats the two pointwise convolutions as equally important, which may be inefficient. Specifically, they suggest that the first layer (expansion) is often over-provisioned, while the second layer (projection)—which is responsible for learning complex feature correlations—is where the real "expressiveness" of the network happens.
Methods: The Asymmetrical Bottleneck
The core innovation is the Asymmetrical Bottleneck Block. Instead of a symmetrical expansion and projection, the authors "tilt" the workload to prioritize the more expressive layers.
The design follows three key principles:
- Reduced Expansion: They shrink the output dimension of the first pointwise convolution.
- Feature Reuse: To compensate for the reduced expansion, they concatenate the original input features directly into the flow (a technique inspired by DenseNet). This ensures the network doesn't lose information despite having fewer parameters in the first layer.
- Migrated Computation: The "saved" computational budget from the first layer is moved to the second pointwise convolution.
They introduce an Asymmetry Rate ($r$) to control this shift. When $r=0$, the block is a standard inverted residual. When $r=1$ (their chosen default), the block becomes asymmetrical. By stacking these blocks, the authors created two main architectures: AsymmNet-L (Large) and AsymmNet-S (Small), using the established MobileNetV3 as a structural baseline to ensure a fair comparison.
Key Results and Insights
AsymmNet was evaluated across a wide variety of benchmarks, including ImageNet classification, face recognition (MS-Celeb-1M), action recognition (HMDB51), and object detection (PASCAL VOC). The architecture consistently outperformed MobileNetV3 (MBV3), particularly at low complexity levels:
- ImageNet Classification: At the 0.35x scale (extremely small), AsymmNet-S achieved 55.0% Top-1 accuracy, a massive 5.2% jump over MBV3 (49.8%) at a similar computational cost (~15M MAdds).
- Action Recognition: On HMDB51, AsymmNet-L reached 52.8% accuracy, outperforming MBV3-L (51.3%) and approaching the performance of much larger models like ResNet50.
- The "Expressiveness" Hypothesis: The results confirmed that the second pointwise convolution is the "MVP" of the bottleneck block. Strengthening this layer while weakening the first (and using feature reuse to bridge the gap) leads to a more efficient use of the total FLOPs budget.
Conclusion
AsymmNet demonstrates that asymmetrical workload distribution is a superior design choice for "ultralight" CNNs within the regime of <220M MAdds. Because the input and output shapes remain the same as standard blocks, AsymmNet can be used as a "drop-in" replacement for existing architectures. While its advantage narrows as model scale increases, it represents a significant step forward for extremely resource-constrained environments. Future work may involve combining this asymmetrical design with Neural Architecture Search (NAS) to further optimize super-lightweight models.
Summary updated