CubeSLAM: Monocular 3D Object Detection and SLAM without Prior Models

Shichao Yang, Sebastian Scherer

Abstract

We present CubeSLAM, a method for single-image 3D cuboid object detection and multi-view object SLAM that operates without prior 3D models. By representing objects as 3D cuboids, we generate high-quality proposals from 2D bounding boxes and vanishing point sampling, which are then refined through edge alignment. These detections are integrated into a multi-view bundle adjustment framework that jointly optimizes camera poses, objects, and points. Our approach demonstrates that 3D object detection and SLAM can mutually benefit each other, providing stronger geometric constraints and scale consistency. Experiments on SUN RGBD and KITTI datasets show significant improvements in both pose estimation and 3D detection accuracy over state-of-the-art monocular SLAM.

Introduction

Traditional monocular SLAM (Simultaneous Localization and Mapping) systems typically rely on tracking low-level features like points and lines. While robust, these systems often suffer from scale drift—where the map's scale changes over time—and lack semantic understanding of the environment. Furthermore, dynamic objects like moving cars are usually treated as outliers and discarded, losing valuable information. CubeSLAM addresses these challenges by incorporating 3D object detection into the SLAM pipeline. By modeling objects as cuboids, the system gains "super-features" that provide long-range geometric constraints, helping to stabilize the map and recover absolute scale.

Single-Image 3D Cuboid Detection

The first component of CubeSLAM is a model-free 3D detector. Starting from a 2D bounding box (generated by standard detectors like YOLO), the system samples vanishing points (VPs) to generate 3D cuboid proposals. Vanishing points represent the intersection of parallel lines in 3D space when projected onto a 2D image plane.

Cuboid proposal generation using vanishing points (VPs) for different visible face configurations.

As shown in the figure above, different configurations of visible faces lead to different cuboid geometries relative to the VPs. These proposals are then scored based on their alignment with detected image edges and a shape prior. This allows the system to detect 3D objects like cars or chairs without needing pre-built CAD models, making it highly flexible for general environments.

Multi-View Object SLAM

The second component integrates these detections into a SLAM framework. We perform a joint optimization called Bundle Adjustment (BA) that includes camera poses, 3D points, and the 3D cuboids themselves as landmarks.

Trajectory estimation and object tracking in a multi-view SLAM scenario showing the benefits of object-level constraints.

A key innovation is the handling of dynamic objects. For moving vehicles, CubeSLAM incorporates a non-holonomic wheel model to estimate trajectories. Instead of ignoring moving cars, the system uses them as additional constraints for camera pose estimation. This multi-view approach not only improves SLAM accuracy but also refines the 3D detections by enforcing temporal consistency across frames.

Results and Key Insights

Evaluations on the KITTI odometry benchmark show that CubeSLAM significantly reduces translation error (down to 1.78%) compared to standard monocular methods. The use of objects as landmarks provides a "lock" on the map scale, preventing the common issue of scale drift. Furthermore, the system remains robust in low-texture environments where point-based tracking often fails, as the cuboid geometry provides a more stable reference than individual points.

Conclusion

CubeSLAM demonstrates a unified approach to 3D perception and localization. By treating objects as integral parts of the SLAM map rather than noise, the system achieves superior accuracy and semantic awareness. Future work will focus on more complex object shapes and improving data association in crowded environments with heavy occlusion.

Summary updated

Abstract

We present a method for single image 3D cuboid object detection and multi-view object SLAM without prior object model, and demonstrate that the two aspects can benefit each other. For 3D detection, we generate high quality cuboid proposals from 2D bounding boxes and vanishing points sampling. The proposals are further scored and selected to align with image edges. Experiments on SUN RGBD and KITTI shows the efficiency and accuracy over existing approaches. Then in the second part, multi-view bundle adjustment with novel measurement functions is proposed to jointly optimize camera poses, objects and points, utilizing single view detection results. Objects can provide more geometric constraints and scale consistency compared to points. On the collected and public TUM and KITTI odometry datasets, we achieve better pose estimation accuracy over the state-of-the-art monocular SLAM while also improve the 3D object detection accuracy at the same time.
BibTeX
@misc{yang2018cubeslam,
  author = {Shichao Yang and Sebastian Scherer},
  title = {{CubeSLAM: Monocular 3D Object Detection and SLAM without Prior Models}},
  year = {2018},
  eprint = {1806.00557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.00557},
}
Figure