TL;DR
SFPN (Synthetic Fusion Pyramid Network) tackles the "scale truncation" problem in object detection, where the standard factor-of-two jump between feature map layers is too abrupt for precise object localization. By inserting "synthetic" intermediate layers to create a smoother scale transition (e.g., 1/2, 1/3, 1/4), the authors significantly boost detection accuracy, particularly for lightweight models like MobileNetV2.
What Problem Does This Paper Address?
In modern object detection, the Feature Pyramid Network (FPN) is a cornerstone. It allows models to detect objects at various sizes by using a hierarchy of feature maps: shallow layers for fine details and deep layers for high-level semantics.
However, most standard backbones (like ResNet or VGG) downsample images using a stride of 2. This creates a "scale gap" where feature maps are sized at 1/2, 1/4, 1/8, and so on. The authors argue this causes scale truncation: two objects of very similar size (e.g., 31x31 pixels vs. 32x32 pixels) might be assigned to entirely different layers of the pyramid. This abrupt transition makes it difficult for the network to fuse features smoothly and accurately represent objects that fall "between" the standard scales.
Methods
The authors introduce a new architecture designed to "fill the gaps" in the feature pyramid:
- Synthetic Fusion Module (SFM): This is the fundamental building block. It takes features from different layers and uses linear scaling (upsampling or downsampling) to create a new intermediate scale. These features are added pixel-wise and processed through a 3x3 convolution.
- Synthetic Fusion Block (SFB): A collection of SFMs organized to integrate features bi-directionally (top-down and bottom-up). The authors stack three of these blocks to perform multiple feature fusions.
- SFPN-X & SOL:
- SFPN-3, 5, 9: These represent the number of output layers. SFPN-3 uses only the original scales, while 5 and 9 add synthetic intermediate layers (like 1/3 or 1/6 scale).
- SOL (Synthetic Output Layers): A strategy where the final prediction head is applied to these new synthetic layers, providing more granular "bins" for objects of different sizes.
Datasets
The models were evaluated on the industry-standard MS-COCO dataset. The backbones used for the experiments (VGG-16, MobileNetV2, and ResNet50) were all pre-trained on ImageNet.
Key Insights and Ideas
- Smoothing the Scale Space: By creating a denser pyramid (1/2, 1/3, 1/4, 1/6...), the network provides a "smoother scale space" for the ground truth to fit into. This reduces the jarring transition between layers.
- Empowering Lightweight Models: One of the most striking findings is that SFPN provides a larger relative performance boost to "weaker" or lightweight models. Because these models have less representational power, the structural help provided by the synthetic layers is more impactful.
- Information Retention: Visualization of confidence maps (heatmaps) shows that adding synthetic layers actually helps the original layers retain more useful information. The synthetic layers act as a buffer that allows the whole pyramid to specialize more effectively.
- Anchor Management: To handle the increased number of output layers in the SOL versions, the authors used k-means clustering to generate more precise anchor boxes, ensuring that the added layers didn't introduce "interference" during training.
Key Results
The authors demonstrated consistent improvements across different backbones and input sizes:
- MobileNetV2 (224x224): The baseline (MobV2-3) achieved 15.5% AP. The version with 9 synthetic output layers (MobV2-9-SOL) jumped to 17.6% AP.
- VGG-16 (320x320): The baseline achieved 20.5% AP, while the SFPN-9-SOL version reached 23.1% AP.
- ResNet50 (224x224): Improved from 20.4% AP to 21.4% AP with the 9-layer synthetic approach.
- Efficiency Trade-off: While accuracy improved, there is a cost in inference speed. For example, MobileNetV2 dropped from 28.72 FPS to 19.23 FPS when moving from the baseline to the 9-layer SOL version. However, the authors note that the gain in accuracy often justifies the moderate drop in speed for many applications.
Limitations and Future Work
- Computational Overhead: Although the authors describe the modules as "lightweight," adding up to six additional layers (in the SFPN-9 version) inevitably increases the parameter count and reduces the Frames Per Second (FPS).
- Manual Design: The current synthetic scales (1/3, 1/6, etc.) are manually defined. Future work could involve using Neural Architecture Search (NAS) to automatically determine the optimal intermediate scales for a specific backbone.
- Anchor Complexity: As the number of output layers increases, the complexity of managing and matching anchors grows, which might make the model harder to tune for specific custom datasets.
Summary updated