Dually Supervised Feature Pyramid for Object Detection and Segmentation

Fan Yang, Cheng Lu, Yandong Guo, Longin Jan Latecki, Haibin LingarXiv 2019 · arXiv

Abstract

Modern object detection and segmentation frameworks rely heavily on Feature Pyramid Networks (FPN) to handle multi-scale objects. However, standard FPNs suffer from gradient degradation in the backbone and task interference in the detection heads. This paper introduces the Dually Supervised Feature Pyramid Network (DSFPN), which incorporates auxiliary prediction heads on the bottom-up path to strengthen supervision and a decoupled head to separate classification and regression tasks. Evaluated on MS COCO, DSFPN provides a consistent performance boost across various backbones with negligible impact on inference speed.

Introduction

Feature Pyramid Networks (FPN) have become a cornerstone of object detection, utilizing a bottom-up backbone for feature extraction and a top-down subnet for semantic enrichment. Despite their success, two primary inefficiencies persist:

  1. Supervision Signal Degradation: Training signals applied only to the top-down path often fade before reaching early backbone layers, leading to sub-optimal feature extraction.
  2. Task Inconsistency: Sharing a single feature space for classification (semantic) and regression (spatial) tasks creates a conflict that hinders the learning of specialized features.

The DSFPN Framework

To resolve these issues, the authors propose two architectural enhancements: Dual Supervision and the Decoupled Head.

Overview of the DSFPN framework. The green path represents the bottom-up backbone with auxiliary supervision, while the blue path shows the top-down feature fusion. The right inset details the decoupled head for classification and regression.

Dual Supervision (DS)

The DS module attaches auxiliary prediction heads to the bottom-up subnet. These heads provide direct "shortcuts" for the loss signal, ensuring that the backbone learns discriminative features from the start. These auxiliary components are used exclusively during training and are discarded during inference, maintaining the model's efficiency.

Decoupled Head (DC)

Standard detection heads use shared fully connected (FC) layers for both classification and regression. DSFPN splits these into two distinct branches, each with its own 1,024-node layers. This separation allows the network to optimize features independently for "what" an object is versus "where" it is located.

Evolution of feature pyramid designs: (a) Featurized image pyramid, (b) Single feature map, (c) SSD-style pyramid, (d) Standard FPN, and (e) The proposed DSFPN.

Experimental Results

The DSFPN was rigorously tested on the MS COCO dataset using various backbones (ResNet-50, ResNet-101, ResNeXt-101).

  • Accuracy Gains: Integrating DSFPN into Faster R-CNN (ResNet-50) yielded a 1.3-point improvement in Average Precision ($AP_{bbox}$). On Cascade Mask R-CNN, it achieved state-of-the-art results with 44.5 $AP_{bbox}$ and 38.1 $AP_{mask}$.
  • Training Efficiency: The additional supervision leads to faster convergence, reaching higher accuracy levels in fewer iterations compared to baseline FPNs. Visualization of feature maps reveals that DSFPN produces cleaner activations with less noise, particularly at lower pyramid levels.
  • Inference Speed: The computational overhead is virtually zero. For Faster R-CNN, inference time only increased from 0.074s to 0.075s, making it highly practical for real-world deployment.

Conclusion

DSFPN demonstrates that simple yet targeted modifications to the supervision flow and head architecture can significantly enhance detector performance. By optimizing the backbone through auxiliary heads and reducing task conflict via decoupling, the framework provides a robust and efficient solution for multi-scale object detection and segmentation. Future work may extend these dual supervision principles to one-stage detectors and other multi-task vision problems.

Summary updated

Abstract

Feature pyramid architecture has been broadly adopted in object detection and segmentation to deal with multi-scale problem. However, in this paper we show that the capacity of the architecture has not been fully explored due to the inadequate utilization of the supervision information. Such insufficient utilization is caused by the supervision signal degradation in back propagation. Thus inspired, we propose a dually supervised method, named dually supervised FPN (DSFPN), to enhance the supervision signal when training the feature pyramid network (FPN). In particular, DSFPN is constructed by attaching extra prediction (i.e., detection or segmentation) heads to the bottom-up subnet of FPN. Hence, the features can be optimized by the additional heads before being forwarded to subsequent networks. Further, the auxiliary heads can serve as a regularization term to facilitate the model training. In addition, to strengthen the capability of the detection heads in DSFPN for handling two inhomogeneous tasks, i.e., classification and regression, the originally shared hidden feature space is separated by decoupling classification and regression subnets. To demonstrate the generalizability, effectiveness, and efficiency of the proposed method, DSFPN is integrated into four representative detectors (Faster RCNN, Mask RCNN, Cascade RCNN, and Cascade Mask RCNN) and assessed on the MS COCO dataset. Promising precision improvement, state-of-the-art performance, and negligible additional computational cost are demonstrated through extensive experiments. Code will be provided.
BibTeX
@misc{yang2019dually,
  author = {Fan Yang and Cheng Lu and Yandong Guo and Longin Jan Latecki and Haibin Ling},
  title = {{Dually Supervised Feature Pyramid for Object Detection and Segmentation}},
  year = {2019},
  eprint = {1912.03730},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.03730},
}
Figure