FAS-Net: Construct Effective Features Adaptively for Multi-Scale Object Detection

Jiangqiao Yan, Yue Zhang, Zhonghan Chang, Tengfei Zhang, Menglong Yan, Wenhui Diao, Hongqi Wang, Xian SunAAAI 2020 · OpenReview

Abstract

Feature pyramid is the mainstream method for multi-scale object detection. In most detectors with feature pyramid, each proposal is predicted based on feature grids pooled from only one feature level, which is assigned heuristically. Recent studies report that the feature representation extracted using this method is sub-optimal, since they ignore the valid information exists on other unselected layers of the feature pyramid. To address this issue, researchers present to fuse valid information across all feature levels. However, these methods can be further improved: the feature fusion strategies, which use common operation (element-wise max or sum) in most detectors, should be replaced by a more flexible way. In this work, a novel method called feature adaptive selection subnetwork (FAS-Net) is proposed to construct effective features for detecting objects of different scales. Particularly, its adaption consists of two level: global attention and local adaptive selection. First, we model the global context of each feature map with global attention based feature selection module (GAFSM), which can strengthen the effective features across each layer adaptively. Then we extract the features of each region of interest (RoI) on the entire feature pyramid to construct a RoI feature pyramid. Finally, the RoI feature pyramid is sent to the feature adaptive selection module (FASM) to integrate the strengthened features according to the input adaptively. Our FAS-Net can be easily extended to other two-stage object detectors with feature pyramid, and supports to analyze the importance of different feature levels for multi-scale objects quantitatively. Besides, FAS-Net can also be further applied to instance segmentation task and get consistent improvements. Experiments on PASCAL07/12 and MSCOCO17 demonstrate the effectiveness and generalization of the proposed method.

Introduction

In modern object detection, the Feature Pyramid Network (FPN) has become the standard architecture for handling objects across various scales. However, standard FPNs rely on a "heuristic assignment" rule: small object proposals are assigned to high-resolution, low-level feature maps, while large objects are assigned to low-resolution, high-level maps.

This paper argues that such a rigid assignment is sub-optimal. A large object may still require fine-grained structural details from lower layers, and a small object may benefit from the high-level semantic context found at the top of the pyramid. While previous architectures like PANet and Libra R-CNN attempted to bridge these levels, they often relied on simple operations like element-wise sums or max-pooling, which lack the flexibility to dynamically select the most relevant information for a specific object.

Comparison of feature pyramid architectures: (a) FPN, (b) Libra R-CNN, (c) PANet, and (d) the proposed FAS-Net.

The FAS-Net Architecture

FAS-Net introduces a more flexible, content-aware approach to feature extraction through two primary modules:

1. Global Attention based Feature Selection Module (GAFSM)

Before region proposals are generated, GAFSM refines the entire feature pyramid. It utilizes a Simplified Non-Local (SNL) block to model global context by analyzing relationships between all pixels in a feature map. This process recalibrates channel importance, effectively strengthening informative features while suppressing noise across all pyramid layers.

2. Feature Adaptive Selection Module (FASM)

FASM is the core innovation of the network. Instead of selecting a single layer for a Region of Interest (RoI), FAS-Net extracts features for that RoI from all levels (P2 through P5). These features are summed and processed through Global Average Pooling to create a descriptor. A small fully-connected network then generates gated weights for each pyramid level, allowing the final RoI representation to be a weighted sum of all levels. To further optimize performance, the authors employ three specialized FASM branches for small, medium, and large proposals.

Detailed structure of FAS-Net highlighting the GAFSM and FASM components.

Experimental Results and Insights

The effectiveness of FAS-Net was validated on the PASCAL VOC and MS COCO 2017 benchmarks:

  • Detection Performance: On PASCAL VOC, FAS-Net improved the baseline Faster R-CNN from 80.48 mAP to 82.18 mAP. On MS COCO, it provided a consistent boost, notably improving small object detection ($AP_S$) by 1.65 points.
  • Instance Segmentation: Integrated into Mask R-CNN, it improved mask AP from 33.62 to 34.69, proving that better detection features directly translate to superior pixel-level masks.
  • Weight Analysis: Quantitative analysis of FASM weights confirmed the authors' hypothesis: large objects naturally assign higher weights (~37%) to top-level (P5) features, while small objects rely more heavily on lower levels (P2 and P3). This demonstrates that the network successfully learns to "order" the best combination of features for specific object sizes.

Conclusion and Future Work

FAS-Net demonstrates that moving beyond heuristic feature selection to an adaptive, attention-based model significantly enhances multi-scale object detection. While the module is highly generalizable—showing gains when integrated with GCNet, DCN, and Libra R-CNN—it does introduce some computational overhead, with inference speed dropping from 22.5 to 17.8 FPS in VOC tests. Future work may focus on automating the currently manual scale division strategy and optimizing the adaptive selection process for real-time applications.

Summary updated

Abstract

Feature pyramid is the mainstream method for multi-scale object detection. In most detectors with feature pyramid, each proposal is predicted based on feature grids pooled from only one feature level, which is assigned heuristically. Recent studies report that the feature representation extracted using this method is sub-optimal, since they ignore the valid information exists on other unselected layers of the feature pyramid. To address this issue, researchers present to fuse valid information across all feature levels. However, these methods can be further improved: the feature fusion strategies, which use common operation (element-wise max or sum) in most detectors, should be replaced by a more flexible way. In this work, a novel method called feature adaptive selection subnetwork (FAS-Net) is proposed to construct effective features for detecting objects of different scales. Particularly, its adaption consists of two level: global attention and local adaptive selection. First, we model the global context of each feature map with global attention based feature selection module (GAFSM), which can strengthen the effective features across each layer adaptively. Then we extract the features of each region of interest (RoI) on the entire feature pyramid to construct a RoI feature pyramid. Finally, the RoI feature pyramid is sent to the feature adaptive selection module (FASM) to integrate the strengthened features according to the input adaptively. Our FAS-Net can be easily extended to other two-stage object detectors with feature pyramid, and supports to analyze the importance of different feature levels for multi-scale objects quantitatively. Besides, FAS-Net can also be further applied to instance segmentation task and get consistent improvements. Experiments on PASCAL07/12 and MSCOCO17 demonstrate the effectiveness and generalization of the proposed method.
BibTeX
@inproceedings{yan2020fas,
  author = {Jiangqiao Yan and Yue Zhang and Zhonghan Chang and Tengfei Zhang and Menglong Yan and Wenhui Diao and Hongqi Wang and Xian Sun},
  title = {{FAS-Net: Construct Effective Features Adaptively for Multi-Scale Object Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=yCHnfcQHXb},
}
Figure