Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification

Hang Xu, Lewei Yao, Zhenguo Li, Xiaodan Liang, Wei Zhang

Abstract

Object detection traditionally relies on handcrafted "neck" and "head" components attached to backbones optimized for classification. Auto-FPN introduces a neural architecture search (NAS) framework to automate the design of these task-specific modules. By employing a differentiable search space with resource constraints, Auto-FPN identifies optimized feature fusion (Auto-fusion) and prediction (Auto-head) structures. Experimental results on benchmarks like COCO demonstrate that Auto-FPN achieves up to a 1.9 mAP improvement over standard FPN baselines while significantly reducing parameter counts and computational costs.

Introduction

In computer vision, object detection is more complex than image classification, requiring both localization and categorization. Most Neural Architecture Search (NAS) research focuses on optimizing "backbones" for classification tasks (e.g., ImageNet) and then transferring them to detection frameworks. However, the components responsible for multi-level feature fusion (the "neck") and final prediction (the "head") are typically generic, handcrafted structures.

The authors of Auto-FPN argue that this "transfer-and-plug" approach is sub-optimal. These task-specific components are rarely tailored to the unique statistics of detection datasets like COCO or BDD. Auto-FPN addresses this by automating the design of the neck and head, ensuring the entire network is optimized for the detection task.

Methodology

Auto-FPN introduces a two-stage automated design process targeting the feature fusion and prediction layers of a detector.

Overview of the Auto-FPN framework, showing the sequential search of the Auto-fusion (neck) and Auto-head modules.

Auto-fusion (The Neck)

The Auto-fusion module searches for optimal paths and operations to combine features from different backbone scales. Unlike fixed structures like FPN or PANet, Auto-fusion explores a search space that includes various dilated convolutions, which are essential for adjusting receptive fields to detect objects of varying sizes.

Comparison of feature fusion architectures. Auto-fusion automates the path search, evolving beyond handcrafted designs like SSD, FPN, and PANet.

Auto-head (The Head)

The Auto-head module optimizes the structure for bounding box regression and classification. While many NAS methods enforce identical "cells" throughout a network, Auto-FPN relaxes this constraint, allowing different cells in the head to possess unique structures tailored to their position in the prediction pipeline.

Technical Innovations

  • Differentiable Search: Utilizing a continuous relaxation of the architecture (similar to DARTS), the framework optimizes network structure via standard gradient descent, ensuring search efficiency.
  • Resource Constraints: To ensure practical deployment, a penalty term for parameter count, FLOPs, and Memory Access Cost (MAC) is incorporated into the loss function.
  • Sequential Search: To manage GPU memory limitations, the neck and head are searched sequentially rather than simultaneously.

Experimental Setup

The framework was evaluated across several diverse benchmarks, including COCO (general detection), Pascal VOC, Berkeley Deep Drive (autonomous driving), Visual Genome (large-scale categories), and ADE20K (scene parsing).

Results and Discussion

Auto-FPN demonstrates significant advantages over manual engineering:

  • Performance and Efficiency: On the COCO dataset, Auto-FPN achieved a +1.9 mAP improvement over the FPN baseline. Notably, the searched modules were 50% smaller in terms of parameters, leading to a 22% reduction in overall model size.
  • Receptive Field Management: Search results consistently favored dilated convolutions in fusion layers, highlighting the importance of receptive field management in detection.
  • Dataset Adaptation: Architectures found for COCO differed from those for BDD, proving that dataset-specific adaptation is superior to a "one-size-fits-all" approach.
  • Transferability: Models searched on COCO transferred effectively to Visual Genome and ADE, outperforming the FPN baseline by 0.2% and 1.6% mAP, respectively.

Conclusion

Auto-FPN proves that automating the design of the "neck" and "head" is as crucial as backbone optimization. While the current framework still relies on ImageNet-pretrained backbones due to computational costs, it sets a precedent for end-to-end automated detector design. Future work may explore more flexible search primitives and extending the search to one-stage detectors more comprehensively.

Summary updated

Abstract

Neural architecture search (NAS) has shown great potential in automating the manual process of designing a good CNN architecture for image classification. In this paper, we study NAS for object detection, a core computer vision task that classifies and localizes object instances in an image. Existing works focus on transferring the searched architecture from classification task (ImageNet) to the detector backbone, while the rest of the architecture of the detector remains unchanged. However, this pipeline is not task-specific or data-oriented network search which cannot guarantee optimal adaptation to any dataset. Therefore, we propose an architecture search framework named Auto-FPN specifically designed for detection beyond simply searching a classification backbone. Specifically, we propose two auto search modules for detection: Auto-fusion to search a better fusion of the multi-level features; Auto-head to search a better structure for classification and bounding-box(bbox) regression. Instead of searching for one repeatable cell structure, we relax the constraint and allow different cells. The search space of both modules covers many popular designs of detectors and allows efficient gradient-based architecture search with resource constraint (2 days for COCO on 8 GPU cards). Extensive experiments on Pascal VOC, COCO, BDD, VisualGenome and ADE demonstrate the effectiveness of the proposed method, e.g. achieving around 5% improvement than FPN in terms of mAP while requiring around 50% fewer parameters on the searched modules.
BibTeX
@inproceedings{xu2019auto,
  author = {Hang Xu and Lewei Yao and Zhenguo Li and Xiaodan Liang and Wei Zhang},
  title = {{Auto-FPN: Automatic Network Architecture Adaptation for Object Detection Beyond Classification}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00675},
}
Figure