SST: Real-time End-to-end Monocular 3D Reconstruction via Sparse Spatial-Temporal Guidance

Chenyangguang Zhang, Zhiqiang Lou, Yan Di, Federico Tombari, Xiangyang Ji

TL;DR

SST is a high-speed neural network that reconstructs 3D scenes from a single moving camera in real-time. By leveraging sparse point clouds from visual SLAM as "geometric hints" and using a novel cross-modal attention mechanism, it achieves state-of-the-art accuracy at a blistering 59 FPS—outperforming previous methods in both detail and speed.

What Problem Does This Paper Address?

Monocular 3D reconstruction—building a 3D model of a room using just a standard smartphone camera—is a "severely ill-posed" problem. Because a single 2D image lacks depth information, computers must infer geometry from multiple viewpoints.

Current "end-to-end" deep learning methods typically struggle with two things:

  1. Fine Details: To save memory, these models often use coarse voxel grids, which "blur out" tiny structures like chair legs or sharp wall corners.
  2. Temporal Fusion: Most models simply average features from different frames. This ignores the fact that some viewpoints are much more informative than others, leading to inconsistent or "mushy" reconstructions.

Methods

The authors introduce SST, which stands for Sparse Spatial-Temporal guidance. The core idea is to stop ignoring the "byproducts" of standard camera tracking.

1. Sparse SLAM Guidance

Most monocular systems already use a Visual SLAM (Simultaneous Localization and Mapping) system to track the camera's position. SLAM produces a "sparse" set of 3D points as it works. SST takes these sparse points and uses them as a geometric skeleton to guide the dense reconstruction.

2. Local Spatial-Temporal Fusion (LSTF)

Instead of just averaging frames, SST uses a Sparse Cross-modal Attention mechanism.

  • Implicit Temporal Attention: The network learns which frames in a sequence are most relevant for a specific 3D area.
  • Explicit Spatial Attention: It uses the sparse SLAM points to "weight" the features, focusing the network's attention on areas where we already have high-confidence geometric hints.

3. Global Spatial-Temporal Fusion (GSTF)

To maintain a consistent map as the camera moves, the authors designed a lightweight, 3D sparse version of a Gated Recurrent Unit (GRU). This allows the system to incrementally update a global 3D model without the massive memory overhead usually associated with high-resolution 3D volumes.

Datasets

The researchers evaluated SST on two industry-standard benchmarks:

  • ScanNet: A large-scale dataset of indoor environments with ground-truth 3D models.
  • 7-Scenes: Used primarily to test how well the model generalizes to new, unseen environments without retraining.

Key Insights and Ideas

  • SLAM is a Goldmine: The paper proves that the "noisy" sparse points generated by traditional SLAM systems are incredibly valuable "priors" that can help deep learning models recover fine details they would otherwise miss.
  • Attention over Averaging: By treating feature fusion as an attention problem rather than a math average, the model can "ignore" redundant or blurry views and "persevere" important geometric cues like corners and edges.
  • Efficiency via Sparsity: By using sparse 3D convolutions (only calculating values where there is actually geometry), they achieved a massive speedup, making the system viable for real-world AR/VR applications.

Key Results

  • Speed: SST runs at 59 FPS on an NVIDIA RTX 3090. For context, this is roughly 8x faster than the high-performing TransFusion model and significantly faster than the previous real-time leader, NeuralRecon (47 FPS).
  • Accuracy: On the ScanNet benchmark, SST achieved the best comprehensive performance, specifically excelling in Precision and F-score.
  • Qualitative Improvements: Visual comparisons show that SST successfully reconstructs "tiny structures" (like the thin legs of a table) that competitors often turn into blobs or omit entirely.

Limitations and Future Work

While SST is a major step forward, some limitations remain:

  • Dependency on SLAM: Because the model relies on sparse points from a SLAM system, if the tracking fails (e.g., due to fast motion or low light), the reconstruction quality will likely degrade.
  • Noise Sensitivity: Sparse SLAM points can be noisy; while the attention mechanism helps filter this, extremely noisy inputs could still introduce artifacts.
  • Future Directions: The authors suggest that this "local-to-global" approach with sparse guidance could be a blueprint for future end-to-end 3D tasks beyond just reconstruction, such as real-time 3D semantic segmentation.

Summary updated

Abstract

Real-time monocular 3D reconstruction is a challenging problem that remains unsolved. Although recent end-to-end methods demonstrate promising results, tiny structures and geometric boundaries are hardly captured due to their insufficient supervision neglecting spatial details and oversimplified feature fusion ignoring temporal cues. To address the problems, we propose an end-to-end 3D reconstruction network SST, which utilizes Sparse estimated points from visual SLAM system as additional Spatial guidance and fuses Temporal features via a cross-modal attention mechanism, achieving more detailed reconstruction results. We propose a Local Spatial-Temporal Fusion module to exploit more informative spatial-temporal cues from multi-view color information and sparse priors, as well a Global Spatial-Temporal Fusion module to refine the local TSDF volumes with the world-frame model from coarse to fine. Extensive experiments on ScanNet and 7-Scenes demonstrate that SST outperforms all state-of-the-art competitors, whilst keeping a high inference speed at 59 FPS, enabling real-world applications with real-time requirements.
BibTeX
@inproceedings{zhang2023sst,
  author = {Chenyangguang Zhang and Zhiqiang Lou and Yan Di and Federico Tombari and Xiangyang Ji},
  title = {{SST: Real-time End-to-end Monocular 3D Reconstruction via Sparse Spatial-Temporal Guidance}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00348},
}

Loading related content...

Figure