Abstract
We present CubeSLAM, a method for single-image 3D cuboid object detection and multi-view object SLAM that operates without prior 3D models. By representing objects as 3D cuboids, we generate high-quality proposals from 2D bounding boxes and vanishing point sampling, which are then refined through edge alignment. These detections are integrated into a multi-view bundle adjustment framework that jointly optimizes camera poses, objects, and points. Our approach demonstrates that 3D object detection and SLAM can mutually benefit each other, providing stronger geometric constraints and scale consistency. Experiments on SUN RGBD and KITTI datasets show significant improvements in both pose estimation and 3D detection accuracy over state-of-the-art monocular SLAM.
Introduction
Traditional monocular SLAM (Simultaneous Localization and Mapping) systems typically rely on tracking low-level features like points and lines. While robust, these systems often suffer from scale drift—where the map's scale changes over time—and lack semantic understanding of the environment. Furthermore, dynamic objects like moving cars are usually treated as outliers and discarded, losing valuable information. CubeSLAM addresses these challenges by incorporating 3D object detection into the SLAM pipeline. By modeling objects as cuboids, the system gains "super-features" that provide long-range geometric constraints, helping to stabilize the map and recover absolute scale.
Single-Image 3D Cuboid Detection
The first component of CubeSLAM is a model-free 3D detector. Starting from a 2D bounding box (generated by standard detectors like YOLO), the system samples vanishing points (VPs) to generate 3D cuboid proposals. Vanishing points represent the intersection of parallel lines in 3D space when projected onto a 2D image plane.
As shown in the figure above, different configurations of visible faces lead to different cuboid geometries relative to the VPs. These proposals are then scored based on their alignment with detected image edges and a shape prior. This allows the system to detect 3D objects like cars or chairs without needing pre-built CAD models, making it highly flexible for general environments.
Multi-View Object SLAM
The second component integrates these detections into a SLAM framework. We perform a joint optimization called Bundle Adjustment (BA) that includes camera poses, 3D points, and the 3D cuboids themselves as landmarks.
A key innovation is the handling of dynamic objects. For moving vehicles, CubeSLAM incorporates a non-holonomic wheel model to estimate trajectories. Instead of ignoring moving cars, the system uses them as additional constraints for camera pose estimation. This multi-view approach not only improves SLAM accuracy but also refines the 3D detections by enforcing temporal consistency across frames.
Results and Key Insights
Evaluations on the KITTI odometry benchmark show that CubeSLAM significantly reduces translation error (down to 1.78%) compared to standard monocular methods. The use of objects as landmarks provides a "lock" on the map scale, preventing the common issue of scale drift. Furthermore, the system remains robust in low-texture environments where point-based tracking often fails, as the cuboid geometry provides a more stable reference than individual points.
Conclusion
CubeSLAM demonstrates a unified approach to 3D perception and localization. By treating objects as integral parts of the SLAM map rather than noise, the system achieves superior accuracy and semantic awareness. Future work will focus on more complex object shapes and improving data association in crowded environments with heavy occlusion.
Summary updated