SFPN: Synthetic FPN for Object Detection

Yu-Ming Zhang, Jun-Wei Hsieh, Chun-Chieh Lee, Kuo-Chin Fan

TL;DR

SFPN (Synthetic Fusion Pyramid Network) tackles the "scale truncation" problem in object detection, where the standard factor-of-two jump between feature map layers is too abrupt for precise object localization. By inserting "synthetic" intermediate layers to create a smoother scale transition (e.g., 1/2, 1/3, 1/4), the authors significantly boost detection accuracy, particularly for lightweight models like MobileNetV2.

What Problem Does This Paper Address?

In modern object detection, the Feature Pyramid Network (FPN) is a cornerstone. It allows models to detect objects at various sizes by using a hierarchy of feature maps: shallow layers for fine details and deep layers for high-level semantics.

However, most standard backbones (like ResNet or VGG) downsample images using a stride of 2. This creates a "scale gap" where feature maps are sized at 1/2, 1/4, 1/8, and so on. The authors argue this causes scale truncation: two objects of very similar size (e.g., 31x31 pixels vs. 32x32 pixels) might be assigned to entirely different layers of the pyramid. This abrupt transition makes it difficult for the network to fuse features smoothly and accurately represent objects that fall "between" the standard scales.

Methods

The authors introduce a new architecture designed to "fill the gaps" in the feature pyramid:

  1. Synthetic Fusion Module (SFM): This is the fundamental building block. It takes features from different layers and uses linear scaling (upsampling or downsampling) to create a new intermediate scale. These features are added pixel-wise and processed through a 3x3 convolution.
  2. Synthetic Fusion Block (SFB): A collection of SFMs organized to integrate features bi-directionally (top-down and bottom-up). The authors stack three of these blocks to perform multiple feature fusions.
  3. SFPN-X & SOL:
    • SFPN-3, 5, 9: These represent the number of output layers. SFPN-3 uses only the original scales, while 5 and 9 add synthetic intermediate layers (like 1/3 or 1/6 scale).
  4. SOL (Synthetic Output Layers): A strategy where the final prediction head is applied to these new synthetic layers, providing more granular "bins" for objects of different sizes.

Datasets

The models were evaluated on the industry-standard MS-COCO dataset. The backbones used for the experiments (VGG-16, MobileNetV2, and ResNet50) were all pre-trained on ImageNet.

Key Insights and Ideas

  • Smoothing the Scale Space: By creating a denser pyramid (1/2, 1/3, 1/4, 1/6...), the network provides a "smoother scale space" for the ground truth to fit into. This reduces the jarring transition between layers.
  • Empowering Lightweight Models: One of the most striking findings is that SFPN provides a larger relative performance boost to "weaker" or lightweight models. Because these models have less representational power, the structural help provided by the synthetic layers is more impactful.
  • Information Retention: Visualization of confidence maps (heatmaps) shows that adding synthetic layers actually helps the original layers retain more useful information. The synthetic layers act as a buffer that allows the whole pyramid to specialize more effectively.
  • Anchor Management: To handle the increased number of output layers in the SOL versions, the authors used k-means clustering to generate more precise anchor boxes, ensuring that the added layers didn't introduce "interference" during training.

Key Results

The authors demonstrated consistent improvements across different backbones and input sizes:

  • MobileNetV2 (224x224): The baseline (MobV2-3) achieved 15.5% AP. The version with 9 synthetic output layers (MobV2-9-SOL) jumped to 17.6% AP.
  • VGG-16 (320x320): The baseline achieved 20.5% AP, while the SFPN-9-SOL version reached 23.1% AP.
  • ResNet50 (224x224): Improved from 20.4% AP to 21.4% AP with the 9-layer synthetic approach.
  • Efficiency Trade-off: While accuracy improved, there is a cost in inference speed. For example, MobileNetV2 dropped from 28.72 FPS to 19.23 FPS when moving from the baseline to the 9-layer SOL version. However, the authors note that the gain in accuracy often justifies the moderate drop in speed for many applications.

Limitations and Future Work

  • Computational Overhead: Although the authors describe the modules as "lightweight," adding up to six additional layers (in the SFPN-9 version) inevitably increases the parameter count and reduces the Frames Per Second (FPS).
  • Manual Design: The current synthetic scales (1/3, 1/6, etc.) are manually defined. Future work could involve using Neural Architecture Search (NAS) to automatically determine the optimal intermediate scales for a specific backbone.
  • Anchor Complexity: As the number of output layers increases, the complexity of managing and matching anchors grows, which might make the model harder to tune for specific custom datasets.

Summary updated

Abstract

FPN (Feature Pyramid Network) has become a basic component of most SoTA one stage object detectors. Many previous studies have repeatedly proved that FPN can caputre better multi-scale feature maps to more precisely describe objects if they are with different sizes. However, for most backbones such VGG, ResNet, or DenseNet, the feature maps at each layer are downsized to their quarters due to the pooling operation or convolutions with stride 2. The gap of downscaling-by-2 is large and makes its FPN not fuse the features smoothly. This paper proposes a new SFPN (Synthetic Fusion Pyramid Network) arichtecture which creates various synthetic layers between layers of the original FPN to enhance the accuracy of light-weight CNN backones to extract objects’ visual features more accurately. Finally, experiments prove the SFPN architecture outperforms either the large backbone VGG16, ResNet50 or light-weight backbones such as MobilenetV2 based on AP score.
BibTeX
@inproceedings{zhang2022sfpn,
  author = {Yu-Ming Zhang and Jun-Wei Hsieh and Chun-Chieh Lee and Kuo-Chin Fan},
  title = {{SFPN: Synthetic FPN for Object Detection}},
  booktitle = {ICIP},
  year = {2022},
  url = {https://openreview.net/forum?id=5ud9_m8FLV},
}

Loading related content...

Figure