Revisiting Efficient Object Detection Backbones from Zero-Shot Neural Architecture Search

Zhenhong Sun, Ming Lin, Xiuyu Sun, Zhiyu Tan, Rong Jin

Abstract

In object detection, the detection backbone consumes more than half of the overall inference cost. Recent researches attempt to reduce this cost by optimizing the backbone architecture with the help of Neural Architecture Search (NAS). However, existing NAS methods for object detection require hundreds to thousands of GPU hours of searching, making them impractical in fast-paced research and development. In this work, we propose a novel zero-shot NAS method to address this issue. The proposed method, named MAE-DET, automatically designs efficient detection backbones via the Maximum Entropy Principle without training network parameters, reducing the architecture design cost to nearly zero yet delivering the state-of-the-art (SOTA) performance. Under the hood, MAE-DET maximizes the differential entropy of detection backbones, leading to a better feature extractor for object detection under the same computational budgets. After merely one GPU day of fully automatic design, MAE-DET innovates SOTA detection backbones on multiple detection benchmark datasets with little human intervention. Comparing to ResNet-50 backbone, MAE-DET is $+2.0\%$ better in mAP when using the same amount of FLOPs/parameters, and is $1.54$ times faster on NVIDIA V100 at the same mAP.

Introduction

In computer vision, the "backbone" is the primary engine of an object detector, often consuming more than half of the total inference cost. While Neural Architecture Search (NAS) has successfully automated the design of these backbones, traditional NAS is computationally expensive, requiring thousands of GPU hours to train and evaluate candidate architectures.

Recently, "zero-shot" NAS methods emerged to solve this by predicting a network's performance without training. However, these methods were primarily designed for image classification and often fail in object detection, which requires the network to recognize objects at many different scales. MAE-DET (Maximum Entropy DETection) addresses this by introducing a zero-shot framework specifically optimized for the multi-scale nature of detection frameworks.

Methodology: Maximum Entropy Detection (MAE-DET)

MAE-DET relies on the Maximum Entropy Principle, which posits that a network’s capacity to represent information is maximized when its differential entropy is at its peak.

Zero-Shot Entropy Estimation

Instead of training, MAE-DET initializes a network with random weights and passes through a single batch of Gaussian noise. It estimates the entropy of the resulting feature maps using a "Gaussian upper bound," which only requires calculating the variance of the features. This allows for performance prediction with zero training cost.

Multi-Scale Entropy Prior (MSEP)

A core innovation of MAE-DET is the Multi-Scale Entropy Prior. Unlike classification-based NAS, which only looks at the final layer, MAE-DET calculates entropy across five different stages of the backbone (C1 through C5). It uses a weighted sum of these entropies to favor the stages most critical for detection heads, such as Feature Pyramid Networks (FPN).

Multi-Scale Entropy Prior (MSEP) for detection backbones.

Search Strategy

The framework employs a coarse-to-fine evolutionary search. It starts with "coarse" mutations (changing block types, depths, and widths) and transitions to "fine" mutations (tweaking only kernel sizes and widths) to polish the final design. By sticking to a standard search space of convolutional blocks, the authors prove that superior performance comes from the optimized structure rather than complex, hand-crafted modules.

Key Results

MAE-DET was evaluated across standard benchmarks including MS COCO, PASCAL VOC, and Cityscapes. The results demonstrate that the searched backbones are both more accurate and significantly faster than standard ResNet architectures:

  • Efficiency: MAE-DET-M achieves a +2.0% mAP improvement over a ResNet-50 backbone on COCO while using the same computational budget (FLOPs and parameters).
  • Speed: On an NVIDIA V100 GPU, the MAE-DET backbone is 1.54x faster than a ResNet-50 at the same accuracy level.
  • Search Cost: The entire search process takes less than 0.6 GPU days and requires less than 2GB of memory, making it highly accessible.
Performance comparison of MAE-DET against ResNet backbones in terms of FLOPs and inference speed.

Conclusion

MAE-DET demonstrates that entropy measured on a randomly initialized network is a powerful proxy for final performance. By aligning the search process with the multi-scale needs of detection heads, the method produces state-of-the-art backbones with nearly zero design cost. Future work may involve extending this Maximum Entropy Principle to Vision Transformer (ViT) architectures and incorporating target data distributions into the zero-shot proxy.

Summary updated

Abstract

In object detection, the detection backbone consumes more than half of the overall inference cost. Recent researches attempt to reduce this cost by optimizing the backbone architecture with the help of Neural Architecture Search (NAS). However, existing NAS methods for object detection require hundreds to thousands of GPU hours of searching, making them impractical in fast-paced research and development. In this work, we propose a novel zero-shot NAS method to address this issue. The proposed method, named MAE-DET, automatically designs efficient detection backbones via the Maximum Entropy Principle without training network parameters, reducing the architecture design cost to nearly zero yet delivering the state-of-the-art (SOTA) performance. Under the hood, MAE-DET maximizes the differential entropy of detection backbones, leading to a better feature extractor for object detection under the same computational budgets. After merely one GPU day of fully automatic design, MAE-DET innovates SOTA detection backbones on multiple detection benchmark datasets with little human intervention. Comparing to ResNet-50 backbone, MAE-DET is $+2.0\%$ better in mAP when using the same amount of FLOPs/parameters, and is $1.54$ times faster on NVIDIA V100 at the same mAP. Code and pre-trained models are available at https://github.com/alibaba/lightweight-neuralarchitecture-search.
BibTeX
@inproceedings{sun2022revisiting,
  author = {Zhenhong Sun and Ming Lin and Xiuyu Sun and Zhiyu Tan and Rong Jin},
  title = {{Revisiting Efficient Object Detection Backbones from Zero-Shot Neural Architecture Search}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=gRaiu4I7RRS},
}
Figure