Abstract
Current state-of-the-art convolutional architectures for object detection are manually designed. Here we aim to learn a better architecture of feature pyramid network for object detection. We adopt Neural Architecture Search and discover a new feature pyramid architecture in a novel scalable search space covering all cross-scale connections. The discovered architecture, named NAS-FPN, consists of a combination of top-down and bottom-up connections to fuse features across scales. NAS-FPN, combined with various backbone models in the RetinaNet framework, achieves better accuracy and latency tradeoff compared to state-of-the-art object detection models. NAS-FPN improves mobile detection accuracy by 2 AP compared to state-of-the-art SSDLite with MobileNetV2 model and achieves 48.3 AP which surpasses Mask R-CNN detection accuracy with less computation time.
Introduction
In object detection, a model must effectively process both fine-grained details for small objects and high-level context for larger ones. Standard architectures utilize a Feature Pyramid Network (FPN) to merge high-resolution, low-semantic features from early layers with low-resolution, high-semantic features from later layers. Traditionally, these pyramids are manually designed, often following a simple top-down pathway. However, the potential design space for connecting these layers is vast. NAS-FPN leverages Neural Architecture Search to automate this design process, discovering a complex web of cross-scale connections that outperforms human-designed intuition.
Methods and Search Space
The authors employ Neural Architecture Search (NAS) to discover a new pyramid topology. The search is centered around a "merging cell"—the fundamental unit of the network. An RNN controller acts as the searcher, making four key decisions to construct each cell:
- Selecting two candidate feature layers from the available pool.
- Selecting a target output resolution.
- Selecting a binary operation to combine them (either Sum or Global Pooling).
- Adding the resulting feature back into the pool for subsequent cells to use.
To ensure the architecture is scalable, the search focuses on a modular structure that can be stacked $N$ times. This modularity allows a single discovered design to be deployed across a wide range of hardware, from fast mobile models (using fewer stacks) to high-accuracy server-side models (using many stacks).
Key Insights
- Complexity Beyond Top-Down: The discovered NAS-FPN architecture is significantly more complex than standard FPNs, featuring a non-linear mix of top-down and bottom-up connections. This suggests that multi-scale feature reuse is more critical than previously thought.
- Backbone Agnostic: A pyramid architecture discovered using a small ResNet-10 backbone transferred exceptionally well to much larger backbones like AmoebaNet and ResNet-101, proving the robustness of the search results.
- Stacking for Scale: The research found that stacking the same discovered pyramid structure multiple times is a more effective scaling strategy than simply increasing the depth or width of a single pyramid.
Results
NAS-FPN was evaluated on the COCO (Common Objects in Context) dataset, demonstrating superior performance across the entire computational spectrum.
On high-end hardware, NAS-FPN combined with an AmoebaNet backbone achieved 48.3 AP, surpassing the performance of Mask R-CNN while requiring less computation time. On mobile devices, using a MobileNetV2 backbone, NAS-FPN improved accuracy by 2 AP over the state-of-the-art SSDLite at equivalent inference latencies. Across various backbones and resolutions, NAS-FPN consistently defines the "Pareto frontier," offering the best accuracy-to-latency tradeoff currently available.
Conclusion
NAS-FPN demonstrates that automated architecture search can significantly improve the "neck" of object detection models. While the initial search process is computationally expensive (requiring 100 TPUs for several days), the resulting architecture is highly efficient and scalable. Future work may involve co-searching for both the backbone and the feature pyramid simultaneously to unlock even greater performance synergies.
Summary updated