Abstract: We present a novel method to reconstruct 3D scenes from images by leveraging deep dense monocular SLAM and fast uncertainty propagation. The proposed approach is able to 3D reconstruct scenes densely, accurately, and in realtime while being robust to extremely noisy depth estimates coming from dense monocular SLAM. Differently from previous approaches, that either use ad-hoc depth filters, or that estimate the depth uncertainty from RGB-D cameras’ sensor models, our probabilistic depth uncertainty derives directly from the information matrix of the underlying bundle adjustment problem in SLAM. We show that the resulting depth uncertainty provides an excellent signal to weight the depth-maps for volumetric fusion. Without our depth uncertainty, the resulting mesh is noisy and with artifacts, while our approach generates an accurate 3D mesh with significantly fewer artifacts. We provide results on the challenging Euroc dataset, and show that our approach achieves 92% better accuracy than directly fusing depths from monocular SLAM, and up to 90% improvements compared to the best competing approach.
Introduction
Reconstructing a 3D scene from a single 2D video stream (monocular SLAM) is a "holy grail" in computer vision. Unlike LiDAR or RGB-D cameras, a standard camera doesn't provide explicit depth measurements. While recent deep-learning models (like DROID-SLAM) have become incredibly good at estimating "dense" depth for every pixel, these estimates are often riddled with noise and artifacts, especially in areas with low texture or repetitive patterns.
Existing solutions usually try to clean up this noise using ad-hoc filters (e.g., "if the depth changes too fast, throw it out") or by assuming a fixed sensor noise model. These methods often fail to distinguish between actual geometry and estimation errors, leading to "messy" 3D models with floating artifacts or distorted surfaces.
Methodology
The researchers propose a pipeline called Probabilistic Volumetric Fusion. Their key innovation is treating the SLAM system not just as a depth estimator, but as a probabilistic engine.
- Backend Uncertainty Extraction: The system uses a dense SLAM backend (specifically DROID-SLAM) which solves a "Bundle Adjustment" (BA) problem—optimizing camera poses and depths simultaneously. The authors show how to efficiently extract the marginal covariances (uncertainty) of the depths from the BA's information matrix.
- Efficient Computation: Calculating these uncertainties for every pixel is usually computationally ruinous. The authors leverage the block-sparse structure of the Hessian matrix and the Schur complement to compute these values rapidly on a GPU.
- Uncertainty Propagation: Since the SLAM system operates on low-resolution depth maps, the authors propagate the uncertainty through a "learned upsampling" layer to match the original image resolution.
- Weighted Volumetric Fusion: Instead of treating every depth estimate as equally valid, they fuse them into a Truncated Signed Distance Function (TSDF) volume where each measurement is weighted by the inverse of its calculated variance.
- Meshing with Bounds: Finally, they extract a 3D mesh (using Marching Cubes) but only for voxels where the uncertainty is below a specific threshold.
Experimental Setup
The approach was evaluated on the EuRoC MAV dataset, a standard benchmark for drone-based SLAM. Specifically, they used the V1 and V2 sequences (recorded in an industrial machine hall), which provide ground-truth 3D point clouds from a laser scanner to measure reconstruction accuracy.
Key Insights and Results
The "math" used to solve SLAM already contains information about how much the system trusts its own estimates. By tapping into the information matrix of the Bundle Adjustment, the authors found a "free" signal that perfectly identifies where the reconstruction is likely to fail. Uncertainty maps clearly highlight that areas like white walls or repetitive "aliased" patterns (like heaters) produce the most noise. By weighting these areas near zero, the final mesh remains clean.
The paper demonstrates a clear Pareto front. By tightening the "uncertainty bound," you get a near-perfect reconstruction of the parts the camera saw clearly, while leaving holes in areas where the data was unreliable. This is often preferable for robotics, where a "wrong" wall is more dangerous than a "missing" one.
Quantitatively, the proposed method achieved 92% better accuracy than simply fusing raw depths from monocular SLAM and showed up to 90% improvement in mapping accuracy compared to Tandem, a leading competing approach. Despite the complex probabilistic math, the entire pipeline runs at 13–15 frames per second on a single NVIDIA RTX 2080 Ti GPU.
Conclusion and Future Work
Because the method aggressively filters uncertain data, the resulting 3D models can have holes in textureless areas (like a plain white ceiling). Additionally, like all monocular systems, the uncertainty bounds are "unitless" unless the system is provided with an external scale (like from an IMU or known object size).
Future work could involve "active exploration"—directing a robot to move specifically toward high-uncertainty areas to "fill in the holes" in the map. There is also potential in combining this probabilistic approach with neural implicit representations (like NeRFs) for even smoother surfaces.
Summary updated