Incremental 3D Cuboid Modeling with Drift Compensation

Masashi Mishima, Hideaki Uchiyama, Diego Thomas, Rin-Ichiro Taniguchi, Rafael Alves Roberto, João Paulo Lima, Veronica Teichrieb

TL;DR

This paper introduces an incremental method for building 3D cuboid models from RGB-D camera data, improving on existing SLAM techniques by adding semantic understanding. It tackles the problem of drift in SLAM by using detected cuboids to refine camera poses and the overall map, leading to more accurate and robust 3D scene reconstruction.

What Problem Does This Paper Address?

Current Simultaneous Localization and Mapping (SLAM) systems, while excellent at building geometric maps of environments (often represented as point clouds), lack semantic understanding. They don't inherently recognize objects or their shapes. This limits their usefulness for tasks requiring object-level manipulation or detailed scene reconstruction, such as creating CAD models. Specifically, existing methods often rely on offline, batch processing, which can be slow and prone to errors in cluttered or dynamic environments. Furthermore, SLAM systems accumulate drift over time, leading to inaccuracies in the map, especially over long trajectories. This paper aims to address these issues by developing an online, incremental method for detecting and modeling cuboids within a SLAM framework, and using these cuboids to correct for SLAM drift.

Methods

The core of the proposed method is an incremental 3D cuboid modeling framework built upon an existing RGB-D SLAM system. Here's a breakdown of the key steps:

  • Incremental Plane Mapping: As the SLAM system generates a point cloud for each frame, planes are incrementally reconstructed and updated. This forms a "plane map" of the environment.
  • Cuboid Detection: Cuboids are detected within the plane map by analyzing the geometric relationships between planes. Specifically, the algorithm looks for sets of three mutually perpendicular planes. It checks for orthogonality, convexity, and proximity to identify potential cuboid faces. The intersection of these three planes is then used to determine the cuboid's position, pose, and size.
  • Incremental Cuboid Update: To improve robustness and reduce false positives, detected cuboid shapes are incrementally updated with new measurements. This sequential update process helps refine the cuboid models over time.
  • Drift Compensation: A novel drift compensation mechanism is introduced. Detected cuboids in the current "local map" are matched with cuboids in a "global cuboid map." If a match is found, the transformation between the local and global cuboid is used to refine the camera pose and the local plane map, effectively correcting for SLAM drift. This process helps merge local maps into the global map more accurately, reducing duplicate objects and improving overall map consistency.
  • Interactive Modeling: An augmented reality (AR)-based interactive system is developed, allowing users to visualize the modeling progress and guide the capture of incomplete cuboids.

Datasets

The paper describes experiments using custom-designed scenes with multiple boxes of varying sizes. These scenes were captured using a Kinect V1 sensor in indoor environments. The ground truth for cuboid sizes was measured using a ruler. The authors note that they created their own dataset because existing datasets for RGB-D sequences with cuboid ground truth annotations were limited.

Key Insights and Ideas

  • Cuboids as Semantic Anchors: The paper highlights the power of using simple geometric primitives like cuboids not just for shape representation but as semantic anchors to improve the overall SLAM process.
  • Incremental Processing for Efficiency: By processing data incrementally frame-by-frame, the system avoids the computational burden of batch processing, making it suitable for real-time applications.
  • Drift Correction via Object Matching: Using detected cuboids to align local and global maps is an elegant way to compensate for SLAM drift, which is a fundamental challenge in long-term mapping. This is more robust than relying solely on geometric registration of point clouds.
  • AR for User Guidance: The AR-based interactive modeling system demonstrates how semantic understanding can directly translate into user-friendly applications, making complex 3D reconstruction tasks more accessible.

Key Results

  • Improved Detection Accuracy: The proposed incremental approach achieved higher precision (0.92) and recall (0.63) compared to a batch-based cuboid detection method (precision 0.64, recall 0.37) in cluttered environments.
  • Effective Drift Compensation: The drift compensation mechanism significantly improved cuboid detection. Without it, five cuboids were missed in a test scene; with drift compensation, the number of detected cuboids increased from 12 to 17, with a notable improvement in precision (0.92 to 0.94) and recall (0.63 to 0.89).
  • Robustness to Arrangement: The method successfully detected cuboids regardless of their arrangement, even with occlusions.
  • Real-time Performance: The computational cost was found to be sufficient for running in a room-scale environment alongside RGB-D SLAM.

Limitations and Future Work

  • Dependency on Plane Detection: The framework relies heavily on accurate plane detection. If planes are not well-detected (e.g., due to noisy point clouds or occlusions), cuboid detection can fail.
  • Requirement for Three Perpendicular Planes: The method requires capturing at least three perpendicular planes to define a cuboid. This can be a limitation when dealing with stacked objects where faces might be hidden or aligned, making it difficult to isolate individual cuboids.
  • Future Work: The authors suggest integrating image features from RGB images to enhance the accuracy and robustness of cuboid detection. They also plan to investigate the framework's performance in more diverse and challenging scenes.

Summary updated

Abstract

This paper presents a framework of incremental 3D cuboid modeling by using the mapping results of an RGB-D camera based simultaneous localization and mapping (SLAM) system. This framework is useful in accurately creating cuboid CAD models from a point cloud in an online manner. While performing the RGB-D SLAM, planes are incrementally reconstructed from a point cloud in each frame to create a plane map. Then, cuboids are detected in the plane map by analyzing the positional relationships between the planes, such as orthogonality, convexity, and proximity. Finally, the position, pose, and size of a cuboid are determined by computing the intersection of three perpendicular planes. To suppress the false detection of the cuboids, the cuboid shapes are incrementally updated with sequential measurements to check the uncertainty of the cuboids. In addition, the drift error of the SLAM is compensated by the registration of the cuboids. As an application of our framework, an augmented reality-based interactive cuboid modeling system was developed. In the evaluation at cluttered environments, the precision and recall of the cuboid detection were investigated, compared with a batch-based cuboid detection method, so that the advantages of our proposed method were clarified.
BibTeX
@inproceedings{mishima2019incremental,
  author = {Masashi Mishima and Hideaki Uchiyama and Diego Thomas and Rin-Ichiro Taniguchi and Rafael Alves Roberto and João Paulo Lima and Veronica Teichrieb},
  title = {{Incremental 3D Cuboid Modeling with Drift Compensation}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=we2c3FnQHg},
}
Figure