Abstract
Inverted bottleneck layers, which are built upon depthwise convolutions, have been the predominant building blocks in state-of-the-art object detection models on mobile devices. In this work, we investigate the optimality of this design pattern over a broad range of mobile accelerators by revisiting the usefulness of regular convolutions. We discover that regular convolutions are a potent component to boost the latency-accuracy trade-off for object detection on accelerators, provided that they are placed strategically in the network via neural architecture search. By incorporating regular convolutions in the search space and directly optimizing the network architectures for object detection, we obtain a family of object detection models, MobileDets, that achieve state-of-the-art results across mobile accelerators. On the COCO object detection task, MobileDets outperform MobileNetV3+SSDLite by 1.7 mAP at comparable mobile CPU inference latencies. MobileDets also outperform MobileNetV2+SSDLite by 1.9 mAP on mobile CPUs, 3.7 mAP on Google EdgeTPU, 3.4 mAP on Qualcomm Hexagon DSP and 2.7 mAP on Nvidia Jetson GPU without increasing latency. Moreover, MobileDets are comparable with the state-of-the-art MnasFPN on mobile CPUs even without using the feature pyramid, and achieve better mAP scores on both EdgeTPUs and DSPs with up to 2x speedup.
Introduction: The FLOPs vs. Latency Paradox
For years, the Inverted Bottleneck (IBN) layer—relying heavily on depthwise convolutions—has been the undisputed king of mobile computer vision. The logic was simple: depthwise convolutions have fewer parameters and lower FLOPs (Floating Point Operations), making them ideal for resource-constrained devices. However, MobileDets points out a critical flaw: low FLOPs do not always equal low latency.
Modern hardware accelerators like Google’s EdgeTPU or Qualcomm’s Hexagon DSP are often optimized for regular (full) convolutions. On an EdgeTPU, a regular convolution can actually run 3x faster than its depthwise counterpart, even if it has 7x more FLOPs. By restricting search spaces to IBN layers only, researchers were leaving significant performance on the table when deploying to anything other than a standard mobile CPU.
Methods: Expanding the Search Space
The researchers proposed expanding the Neural Architecture Search (NAS) search space with two new building blocks inspired by tensor decomposition:
- Fused Inverted Bottleneck (Fused IBN): Instead of a 1x1 expansion followed by a KxK depthwise convolution, this block fuses them into a single KxK regular convolution.
- Tucker Convolution Layers: A generalized bottleneck that uses a 1x1 convolution for input compression, a KxK regular convolution, and another 1x1 convolution for output. This is particularly effective for reducing the cost of large convolutions over high-dimensional feature maps.
They utilized TuNAS, a latency-aware one-shot NAS algorithm, and built hardware-specific cost models to estimate real-world latency during the search process.
NAS revealed that regular convolutions are most beneficial in the early stages of a network. As the network progresses and feature maps become smaller but deeper, the search algorithm tends to switch back to standard IBN layers.
Key Results
MobileDets achieved state-of-the-art (SOTA) results across diverse hardware platforms at comparable latencies. On the COCO object detection task, MobileDets outperformed MobileNetV3+SSDLite by 1.7 mAP on mobile CPUs. The gains on accelerators were even more significant:
- EdgeTPU: Outperformed MobileNetV2+SSDLite by 3.7 mAP.
- DSP: Outperformed MobileNetV2+SSDLite by 3.4 mAP.
- Nvidia Jetson GPU: Outperformed MobileNetV2+SSDLite by 2.7 mAP.
Compared to the state-of-the-art MnasFPN on EdgeTPUs and DSPs, MobileDets achieved better mAP scores with up to a 2x speedup, even without using a complex feature pyramid.
Conclusion
MobileDets demonstrates that the efficiency of a neural layer is hardware-dependent. While CPUs prefer the low-FLOP nature of depthwise convolutions, accelerators crave the high arithmetic intensity of regular convolutions. By strategically re-introducing regular convolutions through NAS, MobileDets provides a family of models that are both faster and more accurate across a wide range of mobile devices. Future work could involve searching for the detection head and the backbone simultaneously to find even tighter hardware integrations.
Summary updated