DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras

Zachary Teed, Jia Deng

Abstract

We introduce DROID-SLAM, a new deep learning based SLAM system. DROID-SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through a Dense Bundle Adjustment layer. DROID-SLAM is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures. Despite training on monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time.

Introduction

Simultaneous Localization and Mapping (SLAM) is a foundational capability for robotics, autonomous vehicles, and AR/VR. Historically, visual SLAM has faced a strict trade-off: classical optimization-based systems (e.g., ORB-SLAM) offer high accuracy but are brittle, often failing due to motion blur or featureless environments. Conversely, recent end-to-end deep learning approaches improve robustness but typically lack the trajectory accuracy of classical methods.

Overview of the DROID-SLAM system, showing input video types and the resulting 3D reconstruction.

DROID-SLAM introduces a novel deep learning architecture that combines recurrent optical flow updates with a differentiable Dense Bundle Adjustment (DBA) layer. This system achieves state-of-the-art performance across monocular, stereo, and RGB-D modalities while demonstrating exceptional zero-shot generalization from synthetic training data to real-world environments.

Methodology

DROID-SLAM builds upon the RAFT architecture for optical flow, adapting it for long-term 3D SLAM. The system maintains a "frame graph" of co-visible images and utilizes two asynchronous threads: a frontend for local tracking and a backend for global optimization and loop closure.

The DROID-SLAM architecture featuring iterative ConvGRU updates and the differentiable Dense Bundle Adjustment (DBA) layer.

The core innovation lies in the iterative update step:

  • Recurrent Updates: For incoming frames, the network extracts features and builds 4D correlation volumes. A ConvGRU iteratively updates a hidden state, predicting revisions to dense optical flow fields and outputting confidence weights.
  • Differentiable Dense Bundle Adjustment (DBA): Flow revisions are passed into a differentiable DBA layer. This layer employs a Gauss-Newton solver to compute updates to camera poses and dense, per-pixel depth maps, ensuring geometric alignment with predicted flow.

By embedding a classical solver directly within the neural network, DROID-SLAM forces the model to respect multi-view geometric constraints, preventing the drift common in pure deep-learning SLAM.

Experiments and Results

The model was trained exclusively on monocular video from the synthetic TartanAir dataset and evaluated across several real-world benchmarks including EuRoC, TUM-RGBD, and ETH3D-SLAM.

Key Performance Metrics

  • TartanAir: Reduced trajectory error by over 60% in both monocular and stereo settings compared to prior state-of-the-art, ranking 1st in the ECCV 2020 SLAM competition.
  • EuRoC: Achieved an average Absolute Trajectory Error (ATE) of 2.2cm in monocular settings, a 43% improvement over ORB-SLAM3.
  • Robustness: The system suffered zero catastrophic failures on TartanAir, EuRoC, and TUM-RGBD. On the ETH3D-SLAM benchmark, it successfully tracked 30 out of 32 datasets, significantly outperforming the next best method (19/32).

Generalization

A key insight is the system's remarkable zero-shot generalization. A single model trained on synthetic monocular video can process real-world stereo and RGB-D data at inference time without retraining. Unlike prior works that use a learned depth basis, DROID-SLAM optimizes dense per-pixel depth directly, preventing overfitting to training-specific depth distributions.

Discussion and Limitations

DROID-SLAM demonstrates that marrying deep learning's robustness with geometric optimization's accuracy is a viable path for next-generation SLAM. However, the system has a significant compute and memory footprint. Global bundle adjustment requires storing feature maps for all keyframes; processing long sequences (up to 5000 frames) necessitates high-end GPUs with 24GB of VRAM. Future research will likely focus on improving memory efficiency and reducing redundant computations to enable deployment on more resource-constrained hardware.

Code is available at: https://github.com/princeton-vl/DROID-SLAM

Summary updated

Abstract

We introduce DROID-SLAM, a new deep learning based SLAM system. DROID-SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through a Dense Bundle Adjustment layer. DROID-SLAM is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures. Despite training on monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time. The URL to our open source code is https://github.com/princeton-vl/DROID-SLAM.
BibTeX
@inproceedings{teed2021droid,
  author = {Zachary Teed and Jia Deng},
  title = {{DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=GXyjZb20TX2i},
}
Figure