SFPN: Synthetic FPN for Object Detection

Yu-Ming Zhang, Jun-Wei Hsieh, Chun-Chieh Lee, Kuo-Chin Fan

Abstract

Feature Pyramid Networks (FPN) have become a fundamental component in state-of-the-art one-stage object detectors, enabling the capture of multi-scale feature maps. However, standard backbones such as VGG, ResNet, and MobileNet typically downsample feature maps by a factor of two at each layer. This discrete jump creates a "scale gap" that can lead to scale truncation, where objects of similar sizes are handled inconsistently across the pyramid. This paper proposes the Synthetic Fusion Pyramid Network (SFPN), an architecture that generates intermediate synthetic layers to smooth the transition between scales. Our experiments demonstrate that SFPN significantly enhances the accuracy of various backbones, with particularly notable gains for lightweight models like MobileNetV2 on the MS-COCO dataset.

Introduction

In modern object detection, the ability to recognize objects across a wide range of sizes is critical. The Feature Pyramid Network (FPN) addresses this by utilizing a hierarchy of feature maps: shallow layers provide high-resolution spatial details for small objects, while deeper layers provide high-level semantic information for larger objects.

Despite its success, the standard FPN architecture suffers from a structural limitation. Because most backbones use a stride of 2 for pooling or convolutions, the resulting feature pyramid is sparse (1/2, 1/4, 1/8, etc.). This creates "scale truncation," where an object measuring 31x31 pixels might be assigned to a different layer than one measuring 32x32 pixels. This abrupt transition makes it difficult for the network to fuse features smoothly. SFPN aims to bridge these gaps by populating the pyramid with synthetic intermediate scales.

Methodology

The SFPN architecture introduces three primary components designed to create a denser and more integrated feature space:

1. Synthetic Fusion Module (SFM)

The SFM is the fundamental building block of the network. It generates intermediate scales by taking features from existing layers and applying linear scaling (upsampling or downsampling) to reach a target intermediate resolution. These features are then combined via pixel-wise addition and refined through a 3x3 convolution to produce a synthetic feature map.

2. Synthetic Fusion Block (SFB)

To ensure that semantic information flows effectively across the pyramid, SFMs are organized into Synthetic Fusion Blocks. These blocks facilitate bi-directional feature integration, allowing both top-down semantic enrichment and bottom-up spatial refinement. In the proposed architecture, three SFBs are typically stacked to maximize feature fusion.

3. SFPN-X and Synthetic Output Layers (SOL)

The authors define different configurations based on the number of output layers:

  • SFPN-3, 5, 9: These denote the total number of layers in the pyramid. While SFPN-3 maintains the original scales, SFPN-5 and SFPN-9 introduce synthetic scales such as 1/3 or 1/6.
  • SOL Strategy: In the Synthetic Output Layer configuration, the final prediction head is applied directly to these new synthetic layers. This provides more granular "bins" for object detection, ensuring that objects are matched to the layer that most closely fits their scale.

Experimental Results

The SFPN was evaluated on the MS-COCO dataset using VGG-16, MobileNetV2, and ResNet50 backbones. The results consistently showed that increasing the density of the feature pyramid leads to higher Average Precision (AP).

  • Lightweight Performance: The most significant relative improvements were observed in lightweight models. MobileNetV2 (224x224) improved from a baseline of 15.5% AP to 17.6% AP using the 9-layer SOL configuration. This suggests that the structural support of synthetic layers compensates for the lower representational power of smaller backbones.
  • Standard Backbones: VGG-16 (320x320) saw an increase from 20.5% AP to 23.1% AP, while ResNet50 (224x224) improved from 20.4% AP to 21.4% AP.
  • Information Retention: Heatmap visualizations indicate that synthetic layers act as a buffer, allowing the original layers to retain more specialized information rather than being forced to represent objects that fall between standard scales.

Efficiency Trade-offs

While accuracy improved, the addition of synthetic layers introduces computational overhead. For MobileNetV2, the inference speed dropped from 28.72 FPS to 19.23 FPS when moving to the 9-layer version. However, for many applications, the ~2% gain in AP justifies this moderate reduction in speed.

Conclusion

SFPN addresses the inherent scale gaps in standard CNN backbones by introducing a synthetic, dense feature pyramid. By smoothing the scale space, the model allows for more precise object localization and better feature fusion. While the current implementation relies on manually defined scales, the significant performance gains—especially for lightweight models—highlight SFPN as a robust architectural improvement for multi-scale object detection.

Summary updated

Abstract

FPN (Feature Pyramid Network) has become a basic component of most SoTA one stage object detectors. Many previous studies have repeatedly proved that FPN can caputre better multi-scale feature maps to more precisely describe objects if they are with different sizes. However, for most backbones such VGG, ResNet, or DenseNet, the feature maps at each layer are downsized to their quarters due to the pooling operation or convolutions with stride 2. The gap of down-scaling-by-2 is large and makes its FPN not fuse the features smoothly. This paper proposes a new SFPN (Synthetic Fusion Pyramid Network) arichtecture which creates various synthetic layers between layers of the original FPN to enhance the accuracy of light-weight CNN backones to extract objects' visual features more accurately. Finally, experiments prove the SFPN architecture outperforms either the large backbone VGG16, ResNet50 or light-weight backbones such as MobilenetV2 based on AP score.
BibTeX
@inproceedings{zhang2022sfpn,
  author = {Yu-Ming Zhang and Jun-Wei Hsieh and Chun-Chieh Lee and Kuo-Chin Fan},
  title = {{SFPN: Synthetic FPN for Object Detection}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=5ud9_m8FLV},
}
Figure