PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN

Yaxu Xie, Jason R. Rambach, Fangwen Shu, Didier Stricker

Abstract

Instance segmentation of planar regions in indoor scenes benefits visual SLAM and other applications such as augmented reality (AR) where scene understanding is required. Existing methods built upon two-stage frameworks show satisfactory accuracy but are limited by low frame rates. In this work, we propose a real-time deep neural architecture that estimates piece-wise planar regions from a single RGB image. Our model employs a variant of a fast single-stage CNN architecture to segment plane instances. Considering the particularity of the target detected, we propose Fast Feature Non-maximum Suppression (FF-NMS) to reduce the suppression errors resulted from overlapping bounding boxes of planes. We also utilize a Residual Feature Augmentation module in the Feature Pyramid Network (FPN). Our method achieves significantly higher frame-rates and comparable segmentation accuracy against two-stage methods. We automatically label over 70,000 images as ground truth from the Stanford 2D-3D-Semantics dataset. Moreover, we incorporate our method with a state-of-the-art planar SLAM and validate its benefits.

Introduction

Identifying planar surfaces—such as walls, floors, and ceilings—is a foundational task for indoor robotics, Augmented Reality (AR), and 3D mapping (SLAM). Traditionally, researchers faced a trade-off between high accuracy and real-time performance. Most current plane estimation methods rely on "two-stage" frameworks like Mask R-CNN, which first propose regions of interest and then perform segmentation. While accurate, these models often operate at approximately 3 FPS, which is insufficient for real-time navigation or smooth AR experiences.

PlaneSegNet breaks this deadlock by introducing a single-stage deep learning architecture. It achieves state-of-the-art accuracy while running at 30.2 FPS—roughly ten times faster than previous heavy-duty models like PlaneRCNN.

Methodology

PlaneSegNet is built upon the YOLACT++ architecture, a fast single-stage segmenter. To optimize it for planar geometry, three key innovations were introduced:

  1. Fast Feature Non-maximum Suppression (FF-NMS): Standard NMS algorithms often delete real planes because they assume high bounding box overlap indicates a duplicate detection. In indoor scenes, planes (like a poster on a wall) naturally overlap. FF-NMS addresses this by analyzing "mask coefficients"—the geometric signature of the shape. If overlapping boxes have distinct mask signatures, the system correctly identifies them as separate planes.
  2. Residual Feature Augmentation (RFA): Standard Feature Pyramid Networks (FPNs) can lose spatial context during data compression. RFA adds a lateral path to re-inject high-resolution spatial information into the prediction layers, improving the boundary accuracy for large structural surfaces.
  3. Automated NDT-RANSAC Labeling: To facilitate large-scale training, the authors developed an automated labeling method using Normal Distribution Transformation (NDT) and RANSAC, generating high-quality ground truth for 70,000 images from the Stanford 2D-3D-S dataset.
The PlaneSegNet architecture, highlighting the Residual Feature Augmentation and the single-stage prediction pipeline.

Experimental Results

The researchers evaluated PlaneSegNet across several benchmarks, including ScanNet, TUM RGB-D, and NYU-V2.

Performance and Accuracy

PlaneSegNet demonstrates a significant leap in efficiency, reaching 30.2 FPS on a standard GPU compared to 3.0 FPS for PlaneRCNN. Beyond speed, it also improves accuracy on structural planes. On the TUM RGB-D dataset, PlaneSegNet achieved a Bounding Box Average Precision ($AP^b$) of 43.97, outperforming PlaneRCNN's 37.13.

Qualitative comparison showing PlaneSegNet's ability to capture large structural planes more accurately than PlaneRCNN.

SLAM Integration

The model's utility was further validated by integrating it into a planar SLAM system (SlamCraft). PlaneSegNet reduced the Root Mean Square Error (RMSE) of trajectories in 5 out of 6 tested sequences, demonstrating that high-quality, real-time plane masks directly contribute to more stable and accurate robot navigation.

Conclusion

PlaneSegNet represents a significant advancement in real-time scene understanding. By shifting from local region proposals to global prototypes and refining suppression logic through FF-NMS, the model achieves the speed necessary for mobile deployment without sacrificing accuracy. While challenges remain in extremely cluttered environments with many small planar surfaces, PlaneSegNet provides a robust foundation for the next generation of real-time AR and robotic applications.

Summary updated

Abstract

Instance segmentation of planar regions in indoor scenes benefits visual SLAM and other applications such as augmented reality (AR) where scene understanding is required. Existing methods built upon two-stage frameworks show satisfactory accuracy but are limited by low frame rates. In this work, we propose a real-time deep neural architecture that estimates piece-wise planar regions from a single RGB image. Our model employs a variant of a fast single-stage CNN architecture to segment plane instances. Considering the particularity of the target detected, we propose Fast Feature Non-maximum Suppression (FF-NMS) to reduce the suppression errors resulted from overlapping bounding boxes of planes. We also utilize a Residual Feature Augmentation module in the Feature Pyramid Network (FPN). Our method achieves significantly higher frame-rates and comparable segmentation accuracy against two-stage methods. We automatically label over 70,000 images as ground truth from the Stanford 2D-3D-Semantics dataset. Moreover, we incorporate our method with a state-of-the-art planar SLAM and validate its benefits.
BibTeX
@inproceedings{xie2021planesegnet,
  author = {Yaxu Xie and Jason R. Rambach and Fangwen Shu and Didier Stricker},
  title = {{PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=AdZXQFmJLXY},
}
Figure