DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras

Zachary Teed, Jia Deng

TL;DR

DROID-SLAM is a deep learning-based SLAM system that achieves state-of-the-art accuracy and robustness across monocular, stereo, and RGB-D sensors. By combining recurrent iterative updates with a novel differentiable Dense Bundle Adjustment (DBA) layer, it significantly outperforms both classical and previous learning-based methods while generalizing remarkably well to unseen real-world environments.

DROID-SLAM Pipeline Overview

What Problem Does This Paper Address?

Simultaneous Localization and Mapping (SLAM) is a foundational challenge in robotics and autonomous driving. Traditionally, the field has been split into two camps:

  1. Classical Methods: These use hand-crafted features and geometric optimization (like Bundle Adjustment). They are highly accurate but brittle, often failing in challenging conditions like low texture or rapid motion.
  2. Learning-Based Methods: These use neural networks to estimate motion and depth. While often more robust to visual noise, they have historically struggled to match the precision of classical geometric solvers on standard benchmarks.

The authors of DROID-SLAM aim to bridge this gap, creating a system that retains the geometric rigor of classical Bundle Adjustment while leveraging the representational power and robustness of deep learning.

Methods

DROID-SLAM introduces a "Differentiable Recurrent Optimization-Inspired Design." Its architecture is built on three main pillars:

DROID-SLAM Architecture and DBA Layer
  • Iterative Updates: Inspired by the RAFT architecture for optical flow, DROID-SLAM uses a Convolutional GRU (Recurrent Neural Network) to iteratively refine camera poses and pixel-wise depth. Unlike RAFT, which only looks at two frames, DROID-SLAM operates on an arbitrary number of frames simultaneously via a frame-graph.
  • Dense Bundle Adjustment (DBA) Layer: This is the core innovation. In each iteration, the network predicts a "flow revision" (how pixels should move). The DBA layer then solves a differentiable least-squares problem to find the pose and depth updates that best match these revisions. This ensures the final output is always geometrically consistent.
  • Two-Tiered Processing: The system runs two asynchronous threads. The Frontend handles real-time tracking and local bundle adjustment, while the Backend performs global bundle adjustment over the entire history of keyframes to minimize long-term drift and handle loop closures.

Datasets

One of the most impressive aspects of DROID-SLAM is its generalization. The model is trained entirely on synthetic monocular video from the TartanAir dataset.

Despite this limited training, it was evaluated across a diverse range of real-world benchmarks:

  • TartanAir (Test Set): Challenging synthetic sequences.
  • EuRoC: Video from a micro aerial vehicle (MAV) in industrial environments.
  • ETH-3D: A benchmark for RGB-D SLAM.
  • TUM-RGBD: Handheld camera sequences with significant motion blur and rolling shutter artifacts.

Key Insights and Ideas

  • Geometry as a Layer: By making the Bundle Adjustment process a differentiable layer within the network, the model learns to "propose" updates that the geometric solver can actually use, effectively learning how to optimize.
  • Dense is Better: Unlike many classical systems that only track a few sparse "points of interest," DROID-SLAM optimizes depth for every single pixel. This dense approach provides more information and leads to much higher robustness.
  • Zero-Shot Generalization: The system demonstrates that a model trained only on monocular data can be applied to stereo or RGB-D data at test time without any retraining. It simply adds the extra sensor data as additional constraints in the DBA layer.
  • Geometric vs. Photometric: DROID-SLAM optimizes geometric error (based on predicted flow) rather than photometric error (pixel brightness). This makes it much more resilient to lighting changes and exposure issues that plague "direct" classical SLAM methods.

Key Results

DROID-SLAM set a new bar for performance across the board:

  • TartanAir: It won the 2020 SLAM competition, reducing error by 62% on the monocular track and 60% on the stereo track compared to the previous best results.
  • EuRoC: On monocular data, it reduced average error by 82% compared to other methods that didn't suffer catastrophic failures.
  • Robustness: On the ETH-3D benchmark, DROID-SLAM successfully tracked 30 out of 32 sequences, whereas the next best system (ORB-SLAM2) only managed 19.
  • Efficiency: Despite its complexity, the system runs in real-time (approx. 20 fps) using a dual-GPU setup (one for the frontend, one for the backend).

Limitations and Future Work

While powerful, DROID-SLAM has some practical hurdles:

  • Hardware Requirements: To achieve real-time performance and handle long trajectories (up to 5000 frames), the system requires significant GPU memory (ideally 24GB).
  • Redundancy: The authors note that the current implementation stores a lot of redundant information in the frame-graph. Future work could involve "culling" or compressing these representations to reduce the memory footprint.
  • Reconstruction Quality: While the system produces a dense 3D map, the paper focuses primarily on the accuracy of the camera trajectory (localization). Evaluating the actual quality of the 3D mesh for mapping purposes remains a separate challenge.

Summary updated

Abstract

We introduce DROID-SLAM, a new deep learning based SLAM system. DROID-SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through a Dense Bundle Adjustment layer. DROID-SLAM is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures. Despite training on monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time. The URL to our open source code is https://github.com/princeton-vl/DROID-SLAM.
BibTeX
@inproceedings{teed2021droid,
  author = {Zachary Teed and Jia Deng},
  title = {{DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=GXyjZb20TX2i},
}
Figure