Temporally-Smooth Global Bundle Adjustment for Real-Time Dense Visual SLAM

Cabrel Wouladje, Golden Tendekai Mumanikidzwa, Md Apon Islam, Huiying Xu, Hongbo Li, Wenzhe Tan, Zhendong Chen, Xinzhong ZhuCVPR 2026 Findings 2026 · OpenReview

TL;DR

This paper introduces a real-time dense visual SLAM system that achieves global consistency by exploiting the temporal coherence of video sequences. By projecting dense geometry into a low-rank temporal basis space, the authors reduce the computational complexity of global bundle adjustment from $O(n^3)$ to $O(n^2k^2)$ (where $k \ll n$). This mathematical reformulation enables full global optimization at 20 FPS on a single GPU, delivering state-of-the-art trajectory accuracy and dense reconstructions without requiring any training data.

What Problem Does This Paper Address?

Simultaneous Localization and Mapping (SLAM) is essential for autonomous navigation. While sparse visual SLAM systems can run in real-time by tracking a limited set of points, dense SLAM—which reconstructs complete surface geometry—is severely bottlenecked by computation.

Jointly optimizing camera poses and millions of 3D points via global Bundle Adjustment (BA) scales cubically ($O(n^3)$), making it intractable for real-time applications. To cope, modern dense SLAM systems typically choose one of two compromises:

  1. Local Windowing: Restricting optimization to a small window of recent keyframes (e.g., DROID-SLAM). This achieves real-time speeds but discards global constraints, leading to tracking drift over time.
  2. Learning-based/Neural Representations: Replacing explicit geometric optimization with learned priors or neural implicit fields (e.g., NICE-SLAM). These methods often sacrifice interpretability, require extensive training data, fail on out-of-distribution scenes, and are typically too slow for real-time deployment (often taking seconds per frame).

This paper challenges the assumption that we must trade global consistency for computational efficiency in dense SLAM.

Methods

The core innovation is a novel temporally-smooth bundle adjustment formulation. The authors observe that because video frames change predictably over time, dense geometry lives on a low-dimensional temporal manifold.

Instead of optimizing millions of independent 3D points, the system represents geometry evolution using a compact set of spatial basis functions ($\Phi$) and temporal coefficients ($\alpha$). Remarkably, just $K \approx 100$ to $200$ basis functions are sufficient to represent millions of 3D points with millimeter accuracy.

Figure 1. System pipeline overview showing: (a) Input RGB video sequence, (b) Initial two-view reconstruction, (c) Temporal basis extraction module with incremental SVD, (d) Global bundle ad- justment in basis space showing the Schur complement structure, and (e) Dense geometry recovery and visualization.

The pipeline operates as follows:

  • Incremental Basis Construction: As new frames arrive, the spatial basis functions are updated on-the-fly using an efficient incremental Singular Value Decomposition (SVD) that runs in $O(K^2M)$ time rather than standard batch SVD.
  • Basis-Space Global BA: The optimization problem is rewritten to solve for camera poses and the temporal coefficients ($\alpha$).
  • Custom Schur Complement Solver: Because the coefficients at a given frame primarily affect neighboring frames, the basis-space Hessian matrix ($H_{\alpha\alpha}$) exhibits a highly sparse, block-diagonal structure. The authors exploit this structure to efficiently eliminate the geometric parameters and solve the reduced system for camera poses.
Figure 2. Visualization of the Schur complement structure showing: (a) Full Hessian H with block structure, (b) Block-diagonal Hαα highlighted, and (c) Reduced Schur complement S showing band-diagonal sparsity pattern. Color coding indicates matrix density and computational bottlenecks.

Datasets

The approach was evaluated across 27 video sequences from three widely accepted standard SLAM benchmarks:

  • TUM RGB-D: 9 sequences featuring indoor environments with motion-capture ground truth.
  • EuRoC MAV: 11 sequences captured from a micro aerial vehicle in industrial environments with laser-scan ground truth.
  • 7-Scenes: 7 challenging indoor environments exhibiting repetitive textures and complex geometry.

No new datasets were introduced in this work.

Key Insights and Ideas

  • The Low-Rank Property of Video: The fundamental "aha moment" is that temporal smoothness in camera motion and scene geometry naturally induces a low-rank structure in the bundle adjustment Hessian.
  • Global is Cheaper than Local: Counter-intuitively, by reformulating the problem in basis space, performing global optimization across all frames becomes computationally cheaper than traditional local windowed methods, while completely eliminating long-term drift.
  • Broad Basins of Attraction: The temporal basis formulation creates a smoother optimization landscape. Using a simple finite-difference warm-start strategy, the optimization converges successfully from the initial guess in over 91% of frames.

Key Results

  • Accuracy: The method achieved a 2.3cm average Absolute Trajectory Error (ATE) on the TUM RGB-D dataset, representing a 41% improvement over DROID-SLAM (3.9cm) and outperforming other baselines like ORB-SLAM3, DSO, and NICE-SLAM.
  • Speed: The system runs 60× faster than existing dense global optimization methods, sustaining a steady 20 FPS on commodity hardware (NVIDIA RTX 5090 GPU + Intel i7 CPU).
  • Scalability: Scalability analysis confirmed near-linear time complexity growth relative to sequence length, keeping per-frame optimization times under 50ms even over thousands of frames.
Figure 3. Cumulative distribution of trajectory accuracy show- ing the percentage of sequences achieving ATE below varying thresholds. Our Temporal-GBA method (green) achieves the low- est AUC of 4.2 cm, demonstrating superior robustness compared to baseline methods.

Limitations and Future Work

Despite its strong performance, the system relies heavily on the assumption of temporal smoothness. As a result, its main limitations include:

  • Abrupt Motion & Non-Rigid Deformations: Performance degrades during rapid, erratic camera shakes or in scenes with significant dynamic, non-rigid movements (e.g., tracking failed on the 7-Scenes stairs sequence due to rapid viewpoint changes).
  • Scene-Dependent Complexity: While simple, structured scenes require fewer bases ($K \approx 85-110$), highly complex or unstructured environments demand more basis functions ($K \approx 250$), increasing the computational load.

Future Directions: The authors point toward developing adaptive basis sizing based on real-time scene statistics, extending the framework to handle dynamic/non-smooth motions, and exploring hybrid approaches that combine geometric temporal bases with learned deep features.

Summary updated

Abstract

We present a real-time dense visual SLAM system that achieves globally consistent reconstruction through a novel temporally-smooth bundle adjustment formulation. Our key contribution is demonstrating that temporal coherence in video sequences induces a low-rank structure in the bundle adjustment Hessian, reducing computational complexity from O(n^3) to O(n^2k^2) where k << n. This enables global optimization at every frame while maintaining 20 FPS performance. We introduce an incremental temporal basis representation that compactly encodes dense geometry evolution, requiring only k ≈ 100 basis functions to represent millions of 3D points. Our optimization jointly minimizes reprojection error and temporal smoothness constraints through a custom Schur complement solver that exploits the block-diagonal structure of basis-space Jacobians. Evaluation on 27 sequences from TUM-RGBD, EuRoC, and 7-Scenes demonstrates state-of-the-art performance: 2.3cm average translation error on TUM RGB-D (41% improvement over DROID-SLAM's 3.9cm), while running 60× faster than existing global optimization methods. Our method outperforms both classical (ORB-SLAM3, DSO) and learning-based (DROID-SLAM, MASt3R-SLAM, NICE-SLAM) approaches in trajectory accuracy while producing globally consistent dense maps.
BibTeX
@inproceedings{wouladje2026temporally,
  author = {Cabrel Wouladje and Golden Tendekai Mumanikidzwa and Md Apon Islam and Huiying Xu and Hongbo Li and Wenzhe Tan and Zhendong Chen and Xinzhong Zhu},
  title = {{Temporally-Smooth Global Bundle Adjustment for Real-Time Dense Visual SLAM}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=lAajRdgFDn},
}
Figure