PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN

Yaxu Xie, Jason R. Rambach, Fangwen Shu, Didier Stricker

In the world of computer vision, identifying "planes"—the flat surfaces like walls, floors, and tables—is a foundational task for indoor robotics, Augmented Reality (AR), and 3D mapping (SLAM). While we’ve become quite good at this, there has always been a frustrating trade-off: you could have high accuracy or high speed, but rarely both.

PlaneSegNet, presented by researchers at DFKI, breaks this deadlock. It introduces a single-stage deep learning architecture that achieves state-of-the-art accuracy while running at a smooth 30 frames per second (FPS)—roughly ten times faster than previous heavy-duty models.

TL;DR

PlaneSegNet is the first real-time, single-stage instance segmentation network specifically designed for piece-wise plane estimation. By optimizing the way features are combined and how overlapping detections are handled, it achieves a 10x speedup over previous models like PlaneRCNN while maintaining superior accuracy on large, structural surfaces.

What Problem Does This Paper Address?

Most current plane estimation methods rely on "two-stage" frameworks (like Mask R-CNN). These models first propose regions of interest and then perform segmentation within those boxes. While accurate, this process is computationally expensive, often limping along at 3 FPS—far too slow for a robot navigating a room or a smooth AR experience on a phone.

Furthermore, planes present a unique geometric challenge. Unlike distinct objects (like a cat or a cup), planes in a 2D image often overlap significantly in their bounding boxes (e.g., a poster on a wall). Standard "Non-maximum Suppression" (NMS) algorithms—the logic used to delete duplicate detections—often accidentally delete real planes because they assume high overlap means a duplicate.

Methods

PlaneSegNet is built upon the YOLACT++ architecture, a single-stage segmenter. The authors introduced three key innovations to tailor it for planes:

  1. Fast Feature Non-maximum Suppression (FF-NMS): Instead of just looking at how much bounding boxes overlap, FF-NMS looks at the "mask coefficients" (the DNA of the shape). If two boxes overlap but their mask signatures are different, the system realizes they are two distinct planes (like a chair back against a wall) and keeps both.
  2. Residual Feature Augmentation (RFA): In standard Feature Pyramid Networks (FPNs), spatial information is often lost when compressing data into fewer channels. RFA adds an extra "lateral" path that injects high-resolution spatial context back into the prediction layers, which is crucial for accurately capturing the boundaries of large walls and floors.
  3. NDT-RANSAC Labeling: To train the model, the authors needed a massive dataset. They developed a new automated labeling method using Normal Distribution Transformation (NDT) combined with RANSAC. This allowed them to automatically generate high-quality plane "ground truth" for 70,000 images from the Stanford 2D-3D-S dataset.

Datasets

The researchers evaluated PlaneSegNet across several standard benchmarks:

  • Stanford 2D-3D-Semantics: Used for large-scale training with the authors' new NDT-RANSAC labels.
  • ScanNet: A common indoor dataset used for cross-validation.
  • TUM RGB-D & NYU-V2: Used to test the model's ability to generalize to new, unseen environments.

Key Insights and Ideas

  • Global vs. Local: Two-stage models crop a "local" area and segment it. PlaneSegNet uses "global" prototypes—it looks at the whole image to create mask templates. This turns out to be much better for large structural planes (walls/floors) because the network doesn't lose the "big picture" context.
  • The Overlap Paradox: In standard object detection, high overlap is a sign of error. In plane detection, high overlap is a structural reality. By shifting the suppression logic from box overlap to feature similarity, the authors solved a major bottleneck in indoor scene understanding.
  • Efficiency is King: By moving to a single-stage architecture, the model bypasses the "re-pooling" steps that slow down two-stage networks, making it viable for real-world deployment on mobile hardware.

Key Results

  • Speed: PlaneSegNet hits 30.2 FPS on a standard GPU, compared to just 3.0 FPS for the previous state-of-the-art, PlaneRCNN.
  • Accuracy: On the TUM RGB-D dataset, PlaneSegNet achieved a Bounding Box Average Precision ($AP^b$) of 43.97, significantly outperforming PlaneRCNN's 37.13.
  • SLAM Integration: When plugged into a SLAM system (SlamCraft), PlaneSegNet reduced the trajectory error (RMSE) in 5 out of 6 tested video sequences, proving that better plane masks lead to more stable robot navigation.

Limitations and Future Work

While PlaneSegNet excels at structural planes, it still faces challenges in highly complex scenes with dozens of tiny, cluttered planar surfaces. The authors also noted that because the masks are assembled globally and then cropped, there can occasionally be "noise" or "leakage" at the edges of the masks.

Future work will likely focus on refining the mask assembly process to reduce this noise and further improving the localization of bounding boxes in cluttered environments.

Summary updated

Abstract

Instance segmentation of planar regions in indoor scenes benefits visual SLAM and other applications such as augmented reality (AR) where scene understanding is required. Existing methods built upon two-stage frameworks show satisfactory accuracy but are limited by low frame rates. In this work, we propose a real-time deep neural architecture that estimates piece-wise planar regions from a single RGB image. Our model employs a variant of a fast single-stage CNN architecture to segment plane instances. Considering the particularity of the target detected, we propose Fast Feature Non-maximum Suppression (FF-NMS) to reduce the suppression errors resulted from overlapping bounding boxes of planes. We also utilize a Residual Feature Augmentation module in the Feature Pyramid Network (FPN) . Our method achieves significantly higher frame-rates and comparable segmentation accuracy against two-stage methods. We automatically label over 70,000 images as ground truth from the Stanford 2D-3D-Semantics dataset. Moreover, we incorporate our method with a state-of-the-art planar SLAM and validate its benefits.
BibTeX
@inproceedings{xie2021planesegnet,
  author = {Yaxu Xie and Jason R. Rambach and Fangwen Shu and Didier Stricker},
  title = {{PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=AdZXQFmJLXY},
}

Loading related content...

Figure