SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos

Yuzheng Liu, Siyan Dong, Shuzhe Wang, Yingda Yin, Yanchao Yang, Qingnan Fan, Baoquan Chen

TL;DR

SLAM3R is a new end-to-end system that achieves high-quality, dense 3D reconstruction from standard monocular RGB video in real-time (20+ FPS). By replacing traditional, slow camera pose optimization with a hierarchical feed-forward neural network, it directly regresses and aligns 3D pointmaps to build globally consistent scene models without ever explicitly calculating camera parameters.

What Problem Does This Paper Address?

Creating a detailed 3D map of a room using just a single moving camera (monocular RGB) is a "holy grail" in computer vision with huge implications for AR/VR and robotics. Traditionally, this is a multi-step process: first, you track the camera's path (SLAM), then you use those paths to triangulate 3D points (Multi-View Stereo).

The problem is that these steps are often disconnected and slow. Traditional methods often require offline processing to get a complete model. Recent "dense SLAM" systems that try to do this online often struggle with a three-way trade-off: they are either accurate but painfully slow (less than 1 FPS), fast but produce sparse/incomplete maps, or they require specialized depth sensors (like LiDAR). SLAM3R aims to break this trade-off by providing accuracy, completeness, and real-time speed using only standard video.

Methods

SLAM3R (pronounced "slæm@r") moves away from explicit geometric optimization. Instead, it uses a two-tier hierarchical framework based on feed-forward neural networks:

  1. Inner-Window Local Reconstruction (I2P): The system processes the video in short, overlapping clips using a sliding window. An Image-to-Points (I2P) network, inspired by DUSt3R, takes these frames and directly predicts a dense 3D pointmap for a "keyframe" within that window. A novel multi-view cross-attention mechanism (illustrated in Figure 3) allows the network to aggregate geometric information from all supporting frames in the window simultaneously.

    Figure 3. Illustration of a decoder block in the proposed keyframe decoder Dkey. We present a minimalist modification to integrate information from different supporting images. Our approach tra- verses each of them, selects its token keys and values, and uses the keyframe queries to interact with them separately across the sup- porting images. This multi-view information is then aggregated through max-pooling. The registration decoder Dreg and scene decoder Dsce (described in Sec. 3.2) share the same architecture.
  2. Inter-Window Global Registration (L2W): To build a full scene, these local pointmaps must be stitched together. The Local-to-World (L2W) network incrementally registers new local reconstructions into a global coordinate system. It uses a retrieval module to find the most relevant previously-seen "scene frames" from a reservoir to act as anchors, ensuring the new points align correctly with the existing global model without needing explicit camera poses.

The entire system overview is shown in Figure 2, highlighting how these two modules work together to produce the final output.

Figure 2. System overview. Given an input monocular RGB video, we apply a sliding window mechanism to convert it into overlapping clips (referred to as windows). Each window is fed into an Image-to-Points (I2P) network to recover 3D points in a local coordinate system. Next, the local points are incrementally fed into a Local-to-World (L2W) network to create a globally consistent scene model. The proposed I2P and L2W networks elegantly share similar architectures. In the I2P step (Sec. 3.1), we select a keyframe as a reference to set up a local coordinate system and use the remaining frames in the window to estimate the 3D geometry captured within it. The points from the first window are used to establish the world coordinate system. We then incrementally fuse the following windows in the L2W step (Sec. 3.2). This process involves retrieving the most relevant already-registered keyframes as a reference, and integrating new keyframes. Through this iterative process, we eventually obtain the full scene reconstruction.

Datasets

The authors trained SLAM3R on a massive combined dataset of approximately 850,000 video clips from:

  • ScanNet++: High-fidelity indoor scene dataset.
  • Aria Synthetic Environments (ASE): Large-scale synthetic indoor data.
  • CO3D-v2: Object-centric real-world video sequences.

For evaluation, they tested on unseen datasets including 7 Scenes (real-world partial scenes), Replica (synthetic complete scenes), ScanNet, Tanks and Temples, and ETH3D.

Key Insights and Ideas

  • Ditching Explicit Poses: The most radical idea is that you don't need to solve for camera rotation and translation to build a 3D map. By regressing pointmaps directly in a shared coordinate space, the network implicitly handles localization.
  • Hierarchical Scaling: While two-view methods like DUSt3R are powerful, they don't scale to long videos. SLAM3R's hierarchy allows it to maintain local detail while managing global consistency through a memory-efficient "buffering set" of scene frames.
  • Confidence-Aware Fusion: The network predicts not just points, but also a confidence map. This allows the system to filter out noisy or uncertain geometry, leading to much cleaner final reconstructions compared to previous end-to-end methods.

Key Results

SLAM3R sets a new state-of-the-art for real-time monocular dense reconstruction:

  • Speed: It runs at 20-25 FPS on a single NVIDIA 4090 GPU, making it truly real-time.
  • Accuracy: On the 7 Scenes dataset, it achieved an average accuracy of 2.13 cm, outperforming both offline optimization methods like DUSt3R (2.19 cm) and other real-time competitors like Spann3R (3.42 cm).
  • Completeness: It consistently produces more complete maps than traditional SLAM systems like DROID-SLAM, which often leave large gaps in the geometry.

Qualitative results in Figure 4 demonstrate that SLAM3R produces reconstructions comparable to high-end offline methods while being significantly faster and more robust to drift than concurrent real-time works.

Figure 4. Qualitative comparison showing SLAM3R's high-quality reconstruction vs other methods.

Limitations and Future Work

Despite its impressive performance, the authors note a few areas for improvement:

  • No Global Bundle Adjustment: Because the system avoids explicit camera parameters, it cannot perform traditional global bundle adjustment to further refine the entire scene at once.
  • Pose Accuracy: While the 3D maps are excellent, the camera poses derived from these maps (via PnP) are currently not as precise as those from SLAM systems specifically designed for tracking.
  • Dynamic Scenes: The current model assumes a static scene; handling moving objects remains a challenge for future iterations.

Summary updated

Abstract

In this paper, we introduce SLAM3R, a novel and effective system for real-time, high-quality, dense 3D reconstruction using RGB videos. SLAM3R provides an end-to-end solution by seamlessly integrating local 3D reconstruction and global coordinate registration through feed-forward neural networks. Given an input video, the system first converts it into overlapping clips using a sliding window mechanism. Unlike traditional pose optimization-based methods, SLAM3R directly regresses 3D pointmaps from RGB images in each window and progressively aligns and deforms these local pointmaps to create a globally consistent scene reconstruction - all without explicitly solving any camera parameters. Experiments across datasets consistently show that SLAM3R achieves state-of-the-art reconstruction accuracy and completeness while maintaining real-time performance at 20+ FPS. Code available at: https://github.com/PKU-VCL-3DV/SLAM3R.
BibTeX
@inproceedings{liu2025slam,
  author = {Yuzheng Liu and Siyan Dong and Shuzhe Wang and Yingda Yin and Yanchao Yang and Qingnan Fan and Baoquan Chen},
  title = {{SLAM3R: Real-Time Dense Scene Reconstruction from Monocular RGB Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01552},
}
Figure