Abstract
Real-time monocular 3D reconstruction is a challenging problem that remains unsolved. Although recent end-to-end methods have demonstrated promising results, tiny structures and geometric boundaries are hardly captured due to their insufficient supervision neglecting spatial details and oversimplified feature fusion ignoring temporal cues. To address the problems, we propose an end-to-end 3D reconstruction network SST, which utilizes Sparse estimated points from visual SLAM system as additional Spatial guidance and fuses Temporal features via a novel cross-modal attention mechanism, achieving more detailed reconstruction results. We propose a Local Spatial-Temporal Fusion module to exploit more informative spatial-temporal cues from multi-view color information and sparse priors, as well a Global Spatial-Temporal Fusion module to refine the local TSDF volumes with the world-frame model from coarse to fine. Extensive experiments on ScanNet and 7-Scenes demonstrate that SST outperforms all state-of-the-art competitors, whilst keeping a high inference speed at 59 FPS, enabling real-world applications with real-time requirements.
Introduction
Monocular 3D reconstruction—the process of building a 3D model of a scene using a standard moving camera—is a "severely ill-posed" problem. Because a single 2D image lacks depth information, systems must infer geometry by integrating information across multiple viewpoints. Current end-to-end deep learning methods often struggle with two primary issues: the loss of fine details due to coarse voxel grids and inconsistent feature fusion that ignores temporal cues.
SST (Sparse Spatial-Temporal guidance) addresses these challenges by leveraging the "byproducts" of standard camera tracking. Most monocular systems already utilize Visual SLAM (Simultaneous Localization and Mapping) to track camera poses, which produces a sparse set of 3D points. SST treats these points as valuable geometric "hints" to guide dense reconstruction, ensuring that fine structures like chair legs or sharp corners are preserved.
Methodology
The SST architecture is divided into three main stages: Data Generation, Local Spatial-Temporal Fusion (LSTF), and Global Spatial-Temporal Fusion (GSTF).
1. Sparse SLAM Guidance
The system extracts sparse 3D points from a traditional SLAM pipeline. These points serve as a geometric skeleton, providing high-confidence spatial priors that guide the network's dense reconstruction process.
2. Local Spatial-Temporal Fusion (LSTF)
Rather than simple feature averaging, SST employs a Sparse Cross-modal Attention mechanism. This includes:
- Implicit Temporal Attention: Learning which frames in a sequence are most relevant for specific 3D regions.
- Explicit Spatial Attention: Using sparse SLAM points to weight features, focusing the network on areas with established geometric evidence.
3. Global Spatial-Temporal Fusion (GSTF)
To maintain map consistency during camera motion, SST uses a lightweight, 3D sparse version of a Gated Recurrent Unit (GRU). This allows for incremental updates to a global 3D model without the prohibitive memory costs of high-resolution 3D volumes.
Key Results
The researchers evaluated SST on the ScanNet and 7-Scenes benchmarks. SST demonstrated significant improvements in both efficiency and reconstruction quality.
- Speed: SST operates at 59 FPS on an NVIDIA RTX 3090, making it approximately 8x faster than TransFusion and notably faster than the previous leader, NeuralRecon (47 FPS).
- Accuracy: On ScanNet, SST achieved state-of-the-art performance, particularly in Precision and F-score.
- Qualitative Detail: Visual comparisons reveal that SST successfully reconstructs thin structures, such as table legs, which competing models often fail to capture or render as indistinct blobs.
Conclusion
SST demonstrates that the "noisy" sparse points from traditional SLAM systems are a goldmine for deep learning models. By treating feature fusion as an attention-based problem and utilizing sparse 3D convolutions, SST achieves a massive speedup while recovering fine details. While the system's performance is tied to the reliability of the underlying SLAM tracking, its "local-to-global" approach provides a robust blueprint for future real-time 3D tasks, such as semantic segmentation in AR/VR environments.
Summary updated