Abstract
Current state-of-the-art convolutional architectures for object detection are manually designed. Here we aim to learn a better architecture of feature pyramid network for object detection. We adopt Neural Architecture Search and discover a new feature pyramid architecture in a novel scalable search space covering all cross-scale connections. The discovered architecture, named NAS-FPN, consists of a combination of top-down and bottom-up connections to fuse features across scales. NAS-FPN, combined with various backbone models in the RetinaNet framework, achieves better accuracy and latency tradeoff compared to state-of-the-art object detection models. NAS-FPN improves mobile detection accuracy by 2 AP compared to state-of-the-art SSDLite with MobileNetV2 model and achieves 48.3 AP which surpasses Mask R-CNN detection accuracy with less computation time.
Introduction
Researchers at Google Brain have utilized Neural Architecture Search (NAS) to move beyond manually designed Feature Pyramid Networks (FPNs) for object detection. By discovering a novel, "atomic" merging cell that can be stacked and scaled, they created NAS-FPN, an architecture that achieves state-of-the-art accuracy while maintaining greater flexibility and speed than traditional models like Mask R-CNN.
The Challenge of Multi-Scale Detection
In object detection, a major challenge is recognizing objects at vastly different scales—from a tiny bird in the distance to a large car in the foreground. To solve this, engineers use Feature Pyramid Networks (FPNs), which combine high-resolution features (essential for localization) with high-level semantic features (essential for classification).
However, most FPNs are designed by hand, typically using simple top-down and lateral connections. Given that the number of ways to connect different feature scales grows exponentially with the number of layers, it is highly unlikely that human intuition has reached the optimal design. The authors set out to automate this design process to find a more efficient and accurate way to fuse multi-scale features.
Methodology
The authors employed Neural Architecture Search (NAS), specifically using a Reinforcement Learning (RL) controller (an RNN trained via Proximal Policy Optimization) to explore a novel search space.
The Merging Cell
The core innovation is the merging cell. Instead of a fixed structure, the NAS algorithm chooses:
- Two input layers from a pool of candidates (which can be from the backbone or previously created cells).
- An output resolution for the new feature.
- A binary operation to combine them: either a simple Sum or Global Pooling (which is more computationally efficient).
Scalability and Stacking
The discovered architecture is "atomic," meaning it can be applied repeatedly. To increase a model's power, one can simply stack the NAS-FPN block $N$ times rather than necessarily increasing the backbone size. This modularity allows the same architecture to scale from lightweight mobile models to heavy-duty server-side detectors, supporting "anytime detection" where a model can exit early after a few blocks if resources are constrained.
Experimental Results
The models were trained and evaluated on the industry-standard COCO (Common Objects in Context) dataset. NAS-FPN demonstrated significant improvements across the board:
- High Performance: NAS-FPN combined with an AmoebaNet backbone achieved 48.3 AP, surpassing the then-state-of-the-art Mask R-CNN while being faster.
- Mobile Superiority: On mobile-sized models (using MobileNetV2), NAS-FPN improved accuracy by 2 AP over the popular SSDLite model at equivalent latency.
Key Insights
- Non-Intuitive Connections: The NAS-discovered architectures often look "messy" compared to human designs, frequently connecting very high-resolution layers directly to very low-resolution ones. These shortcuts proved critical for detecting small objects.
- Backbone Agnosticism: An FPN architecture discovered using a small ResNet-10 backbone transferred effectively to much larger backbones like ResNet-101 or AmoebaNet, suggesting that optimal feature fusion is a universal property.
- Efficiency over Depth: Using NAS-FPN with a lighter backbone often outperformed a standard FPN with a much heavier backbone, proving that the method of feature combination is as vital as the features themselves.
Limitations and Future Work
While the final model is efficient, the process of finding it via NAS is computationally expensive, requiring significant TPU resources. Additionally, this research focused entirely on the FPN; the "heads" of the network (subnets for box and class prediction) remained standard. Future work could involve searching for the entire detector end-to-end. Finally, the irregular structure of the discovered cells can be more difficult to implement and optimize manually compared to the linear structure of a standard FPN.
Summary updated