MobileDets: Searching for Object Detection Architectures for Mobile Accelerators

Yunyang Xiong, Hanxiao Liu, Suyog Gupta, Berkin Akin, Gabriel Bender, Yongzhe Wang, Pieter-Jan Kindermans, Mingxing Tan, Vikas Singh, Bo Chen

Abstract

Inverted bottleneck layers, which are built upon depthwise convolutions, have been the predominant building blocks in state-of-the-art object detection models on mobile devices. In this work, we investigate the optimality of this design pattern over a broad range of mobile accelerators by revisiting the usefulness of regular convolutions. We discover that regular convolutions are a potent component to boost the latency-accuracy trade-off for object detection on accelerators, provided that they are placed strategically in the network via neural architecture search. By incorporating regular convolutions in the search space and directly optimizing the network architectures for object detection, we obtain a family of object detection models, MobileDets, that achieve state-of-the-art results across mobile accelerators. On the COCO object detection task, MobileDets outperform MobileNetV3+SSDLite by 1.7 mAP at comparable mobile CPU inference latencies. MobileDets also outperform MobileNetV2+SSDLite by 1.9 mAP on mobile CPUs, 3.7 mAP on Google EdgeTPU, 3.4 mAP on Qualcomm Hexagon DSP and 2.7 mAP on Nvidia Jetson GPU without increasing latency. Moreover, MobileDets are comparable with the state-of-the-art MnasFPN on mobile CPUs even without using the feature pyramid, and achieve better mAP scores on both EdgeTPUs and DSPs with up to 2x speedup.

The Hardware-Aware Design Shift

For years, the Inverted Bottleneck (IBN) layer—the core of MobileNetV2 and V3—has been the undisputed standard for mobile computer vision. These layers rely on depthwise convolutions to drastically reduce floating-point operations (FLOPs), a winning strategy for mobile CPUs. However, the hardware landscape has shifted toward specialized accelerators like Google’s EdgeTPU and Qualcomm’s Hexagon DSP.

On these platforms, the relationship between FLOPs and actual latency is non-linear. A regular convolution might have significantly more FLOPs than a depthwise counterpart but run faster because the hardware is optimized for dense matrix math. MobileDets challenge the "depthwise-only" dogma by reintroducing regular convolutions into a hardware-aware Neural Architecture Search (NAS) space.

The MobileDet Search Space

The researchers expanded the traditional IBN-only toolkit with two new building blocks designed for accelerator efficiency:

  1. Fused Inverted Bottleneck: This block replaces the 1x1 expansion and KxK depthwise convolution with a single KxK regular convolution. While more "expensive" in FLOPs, it is much more efficient on hardware accelerators.
  2. Tucker Layers: Inspired by tensor decomposition, these layers use a sequence of 1x1, KxK, and 1x1 convolutions to perform flexible channel compression and expansion.

Using TuNAS, a one-shot NAS algorithm, the authors optimized the search directly for object detection (using an SSDLite head) across three target platforms: mobile CPUs, EdgeTPUs, and DSPs.

Architectural Insights from NAS

Through architecture visualization, a key pattern emerged: MobileDets tend to place regular convolutions in the early stages of the network. In these layers, spatial dimensions are large but channel counts are small, making regular convolutions highly efficient. This demonstrates that the placement of operations is as critical as the operations themselves.

Visualization of searched architectures showing the strategic placement of regular convolutions (yellow/pink) in early stages.

Performance on Mobile Accelerators

MobileDets achieved state-of-the-art results by leveraging the expanded search space. The most striking improvements were seen on specialized accelerators. For instance, on DSPs, MobileDets achieved a 3.4 mAP improvement over the MobileNetV2 baseline without increasing latency. The results also highlighted that the search space itself was a primary driver of performance; models searched with the full suite of convolutions (IBN + Fused + Tucker) consistently outperformed those restricted to IBN layers.

COCO mAP vs. DSP Latency trade-off, demonstrating MobileDets' superiority over MobileNetV2 and the benefit of the expanded search space.

Conclusion

MobileDets prove that minimizing FLOPs is not a universal proxy for speed on modern hardware. By re-evaluating regular convolutions through the lens of hardware-aware NAS, this work provides a new family of models that are significantly more efficient for real-world deployment on mobile accelerators. Future work may extend this approach to simultaneously search for both the backbone and detection head, or explore the impact of these architectures on tasks like semantic segmentation.

Summary updated

Abstract

Inverted bottleneck layers, which are built upon depthwise convolutions, have been the predominant building blocks in state-of-the-art object detection models on mobile devices. In this work, we investigate the optimality of this design pattern over a broad range of mobile accelerators by revisiting the usefulness of regular convolutions. We discover that regular convolutions are a potent component to boost the latency-accuracy trade-off for object detection on accelerators, provided that they are placed strategically in the network via neural architecture search. By incorporating regular convolutions in the search space and directly optimizing the network architectures for object detection, we obtain a family of object detection models, MobileDets, that achieve state-of-the-art results across mobile accelerators. On the COCO object detection task, MobileDets outperform MobileNetV3+SSDLite by 1.7 mAP at comparable mobile CPU inference latencies. MobileDets also outperform MobileNetV2+SSDLite by 1.9 mAP on mobile CPUs, 3.7 mAP on Google EdgeTPU, 3.4 mAP on Qualcomm Hexagon DSP and 2.7 mAP on Nvidia Jetson GPU without increasing latency. Moreover, MobileDets are comparable with the state-of-the-art MnasFPN on mobile CPUs even without using the feature pyramid, and achieve better mAP scores on both EdgeTPUs and DSPs with up to 2x speedup. Code and models are available in the TensorFlow Object Detection API: https://github.com/tensorflow/models/tree/master/research/object_detection.
BibTeX
@inproceedings{xiong2021mobiledets,
  author = {Yunyang Xiong and Hanxiao Liu and Suyog Gupta and Berkin Akin and Gabriel Bender and Yongzhe Wang and Pieter-Jan Kindermans and Mingxing Tan and Vikas Singh and Bo Chen},
  title = {{MobileDets: Searching for Object Detection Architectures for Mobile Accelerators}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=L5DV6QbjdUB},
}
Figure