Real-Time Plane Detection with Consistency from Point Cloud Sequences

Jinxuan Xu, Qian Xie, Honghua Chen, Jun Wang

TL;DR

This paper introduces a novel real-time method for detecting consistent planar structures in 3D point cloud sequences from depth sensors. By combining edge-aware superpixel segmentation with a robust plane recovery strategy, it overcomes issues of imprecise boundaries and flickering plane labels common in existing approaches, leading to more stable and accurate plane detection for robotic and computer vision applications.

What Problem Does This Paper Address?

Detecting planar surfaces in 3D point clouds is crucial for many robotic tasks like mapping, localization, and scene understanding. However, existing methods often struggle with real-time performance, precise boundary segmentation, and maintaining consistent plane labels across a sequence of frames. This "flickering" problem, where a plane might be detected in one frame but missed in the next, significantly limits their practical use. The paper aims to address these challenges by developing a method that is both efficient and produces temporally consistent plane detections.

Methods

The proposed method, Real-Time Plane Detection with Consistency (RCPD), operates in three main stages:

  1. Edge-Aware Superpixel Generation: Instead of simple grid-based segmentation, the authors employ an improved K-means clustering that generates superpixels whose boundaries better align with actual object edges. This is achieved by using a novel distance metric that considers both image space and depth space, allowing for more accurate initial segmentation.
  2. Planar Superpixel Identification and Merging: Within each superpixel, Principal Component Analysis (PCA) is used to identify planar regions. These planar superpixels are then merged using a superpixel-wise region growing approach, guided by normal histograms, to form complete, smooth plane structures.
  3. Plane Correspondence Establishment and Recovery: To ensure temporal consistency, a 6D descriptor is introduced to establish one-to-one correspondences between planes in adjacent frames. Crucially, a missing plane recovery strategy is implemented. This strategy leverages the established correspondences and estimated camera motion to re-identify planes that were missed in a particular frame due to sensor instability or other factors.

Datasets

The authors evaluate their method on two publicly available datasets:

  • NYU Dataset: Captured by a Kinect sensor with a resolution of 640x480 pixels.
  • SR4000 Dataset: Contains depth images from a ToF depth camera with a resolution of 176x144 pixels.

The ground truth for these datasets is manually labeled.

Key Insights and Ideas

  • Superpixels for Precision: Using edge-aware superpixels, rather than simple grids, is key to achieving accurate plane boundaries, especially in complex scenes.
  • Depth and Image Space Fusion: Combining geometric information from both image and depth space in the superpixel distance metric leads to more robust segmentation.
  • Temporal Consistency is Crucial: The "flickering" problem is a significant limitation of frame-by-frame detection. Establishing inter-frame correspondences and actively recovering missing planes is essential for practical applications.
  • Recovery Strategy: The proposed plane recovery mechanism, by leveraging motion estimation and relaxed region growing, effectively addresses missed detections without significant computational overhead.

Key Results

  • Single-Frame Performance: The method achieves comparable or better plane detection accuracy than state-of-the-art off-line methods on both NYU and SR4000 datasets, particularly in terms of boundary accuracy.
  • Real-Time Performance: The method is significantly faster than many existing approaches, achieving real-time performance (e.g., 14 ms per frame on NYU dataset) while maintaining high accuracy.
  • Multi-Frame Consistency: In experiments with video sequences, the proposed method dramatically reduces plane label flickering (PFF) and plane missing frequency (PMF) compared to methods like CAPE and CAPE+. For instance, on "scene 2" of the NYU dataset, the PMF was reduced to 0.
  • Ablation Study: The study confirms that smaller superpixel sizes lead to slightly better segmentation accuracy but increase computation time. The plane recovery step is shown to be vital for reducing missing planes and label inconsistencies.

Limitations and Future Work

The authors acknowledge two main limitations:

  • Thin Planes: The method struggles to distinguish very thin planes where depth variations are minimal.
  • Non-Planar Primitives: The current work focuses solely on plane detection; detecting other primitive shapes (like cylinders) is not addressed.

Future work will involve addressing these limitations, particularly by extending the method to detect non-planar primitives.

Summary updated

BibTeX
@inproceedings{xu2021real,
  author = {Jinxuan Xu and Qian Xie and Honghua Chen and Jun Wang},
  title = {{Real-Time Plane Detection with Consistency from Point Cloud Sequences}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=r7sQIeLCVWU},
}
Figure