Abstract
Instance segmentation of planar regions in indoor scenes benefits visual SLAM and other applications such as augmented reality (AR) where scene understanding is required. Existing methods built upon two-stage frameworks show satisfactory accuracy but are limited by low frame rates. In this work, we propose a real-time deep neural architecture that estimates piece-wise planar regions from a single RGB image. Our model employs a variant of a fast single-stage CNN architecture to segment plane instances. Considering the particularity of the target detected, we propose Fast Feature Non-maximum Suppression (FF-NMS) to reduce the suppression errors resulted from overlapping bounding boxes of planes. We also utilize a Residual Feature Augmentation module in the Feature Pyramid Network (FPN). Our method achieves significantly higher frame-rates and comparable segmentation accuracy against two-stage methods. We automatically label over 70,000 images as ground truth from the Stanford 2D-3D-Semantics dataset. Moreover, we incorporate our method with a state-of-the-art planar SLAM and validate its benefits.
Introduction
Identifying planar surfaces—such as walls, floors, and ceilings—is a foundational task for indoor robotics, Augmented Reality (AR), and 3D mapping (SLAM). Traditionally, researchers faced a trade-off between high accuracy and real-time performance. Most current plane estimation methods rely on "two-stage" frameworks like Mask R-CNN, which first propose regions of interest and then perform segmentation. While accurate, these models often operate at approximately 3 FPS, which is insufficient for real-time navigation or smooth AR experiences.
PlaneSegNet breaks this deadlock by introducing a single-stage deep learning architecture. It achieves state-of-the-art accuracy while running at 30.2 FPS—roughly ten times faster than previous heavy-duty models like PlaneRCNN.
Methodology
PlaneSegNet is built upon the YOLACT++ architecture, a fast single-stage segmenter. To optimize it for planar geometry, three key innovations were introduced:
- Fast Feature Non-maximum Suppression (FF-NMS): Standard NMS algorithms often delete real planes because they assume high bounding box overlap indicates a duplicate detection. In indoor scenes, planes (like a poster on a wall) naturally overlap. FF-NMS addresses this by analyzing "mask coefficients"—the geometric signature of the shape. If overlapping boxes have distinct mask signatures, the system correctly identifies them as separate planes.
- Residual Feature Augmentation (RFA): Standard Feature Pyramid Networks (FPNs) can lose spatial context during data compression. RFA adds a lateral path to re-inject high-resolution spatial information into the prediction layers, improving the boundary accuracy for large structural surfaces.
- Automated NDT-RANSAC Labeling: To facilitate large-scale training, the authors developed an automated labeling method using Normal Distribution Transformation (NDT) and RANSAC, generating high-quality ground truth for 70,000 images from the Stanford 2D-3D-S dataset.
Experimental Results
The researchers evaluated PlaneSegNet across several benchmarks, including ScanNet, TUM RGB-D, and NYU-V2.
Performance and Accuracy
PlaneSegNet demonstrates a significant leap in efficiency, reaching 30.2 FPS on a standard GPU compared to 3.0 FPS for PlaneRCNN. Beyond speed, it also improves accuracy on structural planes. On the TUM RGB-D dataset, PlaneSegNet achieved a Bounding Box Average Precision ($AP^b$) of 43.97, outperforming PlaneRCNN's 37.13.
SLAM Integration
The model's utility was further validated by integrating it into a planar SLAM system (SlamCraft). PlaneSegNet reduced the Root Mean Square Error (RMSE) of trajectories in 5 out of 6 tested sequences, demonstrating that high-quality, real-time plane masks directly contribute to more stable and accurate robot navigation.
Conclusion
PlaneSegNet represents a significant advancement in real-time scene understanding. By shifting from local region proposals to global prototypes and refining suppression logic through FF-NMS, the model achieves the speed necessary for mobile deployment without sacrificing accuracy. While challenges remain in extremely cluttered environments with many small planar surfaces, PlaneSegNet provides a robust foundation for the next generation of real-time AR and robotic applications.
Summary updated