SPRNet: Single-Pixel Reconstruction for One-Stage Instance Segmentation

Jun Yu, Jinghan Yao, Jian Zhang, Zhou Yu, Dacheng Tao

TL;DR

This paper introduces SPRNet, a novel one-stage instance segmentation framework that achieves comparable accuracy to two-stage methods like Mask R-CNN but with significantly higher inference speeds. It achieves this by reconstructing instance masks from individual pixels within a feature map, bypassing the computationally expensive region proposal step.

What Problem Does This Paper Address?

Object instance segmentation, a crucial task in computer vision that involves identifying and delineating each distinct object instance in an image, is computationally demanding. Existing state-of-the-art methods often rely on two-stage approaches, which first propose regions of interest (RoIs) and then perform segmentation within those regions. While effective, this two-stage process, particularly the Region Proposal Network (RPN), becomes a bottleneck, limiting real-time applications. The paper aims to bridge this gap by developing an efficient one-stage instance segmentation method that offers both speed and accuracy.

Methods

The core innovation of SPRNet is its Single Pixel Reconstruction (SPR) branch, which is integrated into existing one-stage object detectors (like RetinaNet). Instead of relying on RoIs, SPRNet reconstructs an instance mask from each pixel in the convolutional feature map. This is achieved through a cascade of deconvolution layers that progressively build a mask prediction.

To enhance the feature representation for this pixel-wise reconstruction, the authors introduce Gate-FPN (GFPN). GFPN is a modification of the Feature Pyramid Network (FPN) that addresses two key issues:

  1. Spatial Shift: FPN's simple element-wise summation of feature maps from different levels can introduce spatial shifts from higher levels, potentially corrupting fine-grained spatial information in lower levels.
  2. Gradient Propagation: The summation operation can lead to uncontrolled gradient propagation, where lower-level features might dominate the learning process, hindering the independent learning of features at different scales.

GFPN uses a gating mechanism to adaptively control the information flow between feature levels, allowing the network to learn which features are most relevant for each level. This gating mechanism uses shared separable convolutions and sigmoid activations to generate score maps that weigh the contribution of different feature maps.

The SPRNet architecture consists of:

  • A backbone network (e.g., ResNet-50).
  • A GFPN for multi-scale feature fusion.
  • Three parallel branches: a classification branch, a bounding box regression branch (similar to RetinaNet), and the novel mask prediction branch.

The mask branch uses multi-scale fusion with dilated convolutions to capture rich morphological information at each pixel. It then employs a series of deconvolutions to reconstruct a 32x32 mask score map from a single pixel. This mask is then resized to fit the predicted bounding box.

Datasets

The experiments were conducted on the MS-COCO dataset. Specifically, the authors used the MS-COCO 2017 train dataset for training and the 5k validation images for reporting results and ablations.

Key Insights and Ideas

  • One-Stage Instance Segmentation is Feasible: The paper demonstrates that it's possible to achieve competitive instance segmentation performance without the overhead of region proposals, paving the way for faster and more efficient systems.
  • Pixel as Instance Carrier: The core idea of using individual pixels as "instance carriers" to reconstruct masks is a novel and effective approach for one-stage segmentation.
  • GFPN for Smarter Feature Fusion: The Gate-FPN mechanism is crucial for improving feature fusion by adaptively controlling information flow and gradient propagation, leading to better detection and segmentation. This addresses fundamental limitations of traditional FPN.
  • Trade-offs in Thresholding: The choice of overlap threshold for positive anchor sampling during training significantly impacts performance, with a 0.7 threshold found to offer a good balance between robustness and training ease.

Key Results

  • Comparable Accuracy, Higher Speed: SPRNet with a ResNet-50 backbone achieves comparable mask AP to Mask R-CNN (ResNet-50-FPN) while running significantly faster (9 fps vs. 7 fps on GTX 1080Ti).
  • Improved Detection: SPRNet with GFPN shows all-round improvements on box AP compared to RetinaNet with FPN, particularly for small objects.
  • GFPN Benefits: GFPN consistently outperforms FPN in both detection and segmentation tasks, showing improvements in AP across various scales. For instance, GFPN provides up to a 4.3% improvement in AP for large object detection due to its gradient blocking capabilities.
  • Robustness: The method demonstrates good capability in handling challenging scenarios like crowded objects and objects with uncommon morphology.

Limitations and Future Work

  • Mask Detail vs. Recall: While SPRNet achieves higher recall (detecting more objects), Mask R-CNN provides more accurate mask boundaries for detected objects, especially in extreme situations. The authors acknowledge that recovering very detailed and accurate masks from single pixels is challenging.
  • Regression Shift Impact: The accuracy of the final mask prediction can be significantly influenced by the accuracy of the bounding box regression, especially when boxes are mis-aligned.
  • Input Scale Sensitivity: The paper notes that existing ResNet backbones may struggle with very small or very large objects when input scales are too extreme (e.g., <400 or >800 pixels), limiting the ability to extract spatial information.

Future work could explore further improvements in mask detail recovery and investigate more robust methods for handling bounding box regression inaccuracies. The authors also suggest that the SPRNet framework is general and can be extended to other one-stage detectors.

Summary updated

Abstract

Object instance segmentation is one of the most fundamental but challenging tasks in computer vision, and it requires the pixel-level image understanding. Most existing approaches address this problem by adding a mask prediction branch to a two-stage object detector with the region proposal network (RPN). Although producing good segmentation results, the efficiency of these two-stage approaches is far from satisfactory, restricting their applicability in practice. In this article, we propose a one-stage framework, single-pixel reconstruction net (SPRNet), which performs efficient instance segmentation by introducing a single-pixel reconstruction (SPR) branch to off-the-shelf one-stage detectors. The added SPR branch reconstructs the pixel-level mask from every single pixel in the convolution feature map directly. Using the same ResNet-50 backbone, SPRNet achieves comparable mask AP with Mask R-CNN at a higher inference speed and gains all-round improvements on box AP at every scale compared with RetinaNet.
BibTeX
@article{yu2021sprnet,
  author = {Jun Yu and Jinghan Yao and Jian Zhang and Zhou Yu and Dacheng Tao},
  title = {{SPRNet: Single-Pixel Reconstruction for One-Stage Instance Segmentation}},
  journal = {IEEE Trans. Cybernetics},
  year = {2021},
  url = {https://openreview.net/forum?id=TmrgoRrXzjm},
}
Figure