Single Pixel Reconstruction for One-stage Instance Segmentation

Jinghan Yao, Zhou Yu, Jun Yu, Dacheng Tao

TL;DR

This paper introduces SPRNet, a novel one-stage instance segmentation framework that reconstructs object masks from individual pixels, achieving comparable accuracy to two-stage methods like Mask R-CNN but with significantly higher inference speed. It also proposes Gate-FPN (GFPN) to improve feature fusion in one-stage detectors, leading to better detection and segmentation performance across various object scales.

What Problem Does This Paper Address?

Object instance segmentation, a crucial task in computer vision for understanding images at a pixel level, has traditionally been dominated by two-stage approaches. These methods, which typically involve a Region Proposal Network (RPN) followed by a mask prediction branch, achieve good results but suffer from slow inference speeds, limiting their real-world applicability. The challenge is to develop a method that can perform instance segmentation efficiently (one-stage) while maintaining high accuracy, a feat that has been difficult to achieve without the explicit region proposals used in two-stage methods.

Methods

SPRNet is a one-stage instance segmentation framework built upon existing one-stage object detectors like RetinaNet. The core innovation is the Single Pixel Reconstruction (SPR) branch. Instead of relying on region proposals, SPRNet reconstructs the instance mask directly from each pixel in the convolutional feature map. This is achieved through a decoder-like structure that uses a series of deconvolution layers to gradually build a mask from a single pixel.

To enhance the feature representation for this task, the authors introduce Gate-FPN (GFPN). This is a modification of the Feature Pyramid Network (FPN) that addresses issues with traditional FPN's feature fusion. GFPN uses a gating mechanism to adaptively control the information flow between different feature levels, preventing spatial shifts and mitigating gradient propagation problems that can weaken feature learning. This leads to improved feature quality for both detection and segmentation.

Datasets

The experiments were conducted on the MS-COCO 2017 dataset.

Key Insights and Ideas

  • One-stage instance segmentation is feasible and efficient: The paper demonstrates that it's possible to achieve competitive instance segmentation performance without the overhead of region proposal networks, leading to significant speedups.
  • Pixel-level reconstruction is a viable alternative: Reconstructing masks from individual pixels, rather than from proposed regions, is an effective strategy for one-stage segmentation.
  • Gating for better feature fusion: The proposed Gate-FPN (GFPN) effectively addresses limitations in traditional FPN by adaptively controlling feature fusion, leading to improved performance across different object scales.
  • Improved handling of object scales: GFPN's ability to better manage gradient flow helps in detecting both small and large objects more effectively.

Key Results

  • Comparable accuracy to Mask R-CNN: SPRNet with a ResNet-50 backbone achieves comparable mask AP to Mask R-CNN.
  • Significantly higher inference speed: SPRNet runs at 9 fps, while Mask R-CNN runs at 7 fps on the same hardware (GTX 1080Ti), representing a ~30% speedup.
  • Improved detection performance: SPRNet with GFPN shows all-round improvements on box AP at every scale compared to RetinaNet with FPN.
  • Enhanced recall: SPRNet demonstrates higher recall than Mask R-CNN, meaning it detects more objects.
  • Robustness to overlapping objects: The method shows promising results in distinguishing borders and maintaining edge consistency for overlapped objects.

Limitations and Future Work

  • Instance detail vs. recall: While SPRNet achieves higher recall, Mask R-CNN still exhibits better performance in capturing fine instance details, particularly in extreme situations. This is attributed to Mask R-CNN's region-proposal-based approach which allows for more precise segmentation of individual regions.
  • Regression shift impact: The paper acknowledges that in a one-stage framework, regression errors can lead to misaligned masks, which is a challenge for recovering very detailed and accurate masks from pixels.
  • Future work: The authors suggest exploring further improvements to the mask reconstruction process to achieve even finer details and potentially integrating SPRNet with other one-stage detectors. The effectiveness of GFPN also suggests its potential application in other computer vision tasks.

Summary updated

Abstract

Object instance segmentation is one of the most fundamental but challenging tasks in computer vision, and it requires the pixel-level image understanding. Most existing approaches address this problem by adding a mask prediction branch to a two-stage object detector with the Region Proposal Network (RPN). Although producing good segmentation results, the efficiency of these two-stage approaches is far from satisfactory, restricting their applicability in practice. In this paper, we propose a one-stage framework, SPRNet, which performs efficient instance segmentation by introducing a single pixel reconstruction (SPR) branch to off-the-shelf one-stage detectors. The added SPR branch reconstructs the pixel-level mask from every single pixel in the convolution feature map directly. Using the same ResNet-50 backbone, SPRNet achieves comparable mask AP to Mask R-CNN at a higher inference speed, and gains all-round improvements on box AP at every scale comparing with RetinaNet.
BibTeX
@article{yao2021single,
  author = {Jinghan Yao and Zhou Yu and Jun Yu and Dacheng Tao},
  title = {{Single Pixel Reconstruction for One-stage Instance Segmentation}},
  journal = {IEEE Trans. Cybernetics},
  year = {2021},
  url = {https://openreview.net/forum?id=TmrgoRrXzjm},
}
Figure