Multi-Session SLAM with Differentiable Wide-Baseline Pose Optimization

Lahav Lipson, Jia Deng

Introduction

Standard Simultaneous Localization and Mapping (SLAM) systems are typically designed for single, continuous video streams. Tracking failures caused by rapid motion, occlusion, or camera cuts often lead to map fragmentation. Multi-session SLAM addresses this by stitching disjoint video sequences into a unified global map. A primary challenge in monocular setups is the "7-DOF gauge freedom"—the need to estimate relative rotation, translation, and scale ($Sim(3)$) between separate sessions to ensure a consistent global geometry.

Methods: Iterative Differentiable Optimization

The proposed system utilizes a unified backbone for both local visual odometry and global wide-baseline matching. The architecture centers on an iterative update operator, which uses a Recurrent Neural Network (RNN) to refine optical flow and camera poses through multiple passes.

The iterative architecture refines feature matches and camera poses over multiple iterations using a deep backbone.

Differentiable SED Solver

A key innovation is the introduction of a differentiable solver layer that minimizes the Symmetric Epipolar Distance (SED). By making the geometric solver differentiable, the network can be trained end-to-end. This allows the model to learn "confidence weights" for every match; if a match violates geometric constraints, the gradients from the final pose error teach the network to down-weight that feature in future iterations. This mechanism effectively ignores dynamic objects or textureless regions that would otherwise degrade pose accuracy.

Two-Stage Pose Estimation

To handle the wide baselines typical of multi-session alignment, the system employs a two-stage strategy. A robust, weighted 8-point algorithm first provides a "pre-conditioned" rough estimate. This initialization places the pose within the "basin of convergence" for the non-linear SED solver, which then performs high-precision refinement to find the optimal alignment.

The two-stage pipeline combines preconditioning with a differentiable SED solver to ensure robust convergence across wide baselines.

Evaluation and Results

The model was trained on synthetic homographies and fine-tuned on datasets like Scannet and Megadepth, with the VO backbone trained on TartanAir. Evaluation across standard benchmarks demonstrated significant improvements over existing state-of-the-art methods:

  • SLAM Performance: On the EuRoC-MAV dataset, the system significantly outperformed ORB-SLAM3, reducing the Root Mean Square Error (RMSE) by up to 11x in challenging sequences.
  • Robustness: In the ETH3D benchmark, the method successfully processed challenging scenes (e.g., "Sofa" and "Plant") where standard industry systems failed to initialize or lost track.
  • Wide-Baseline Matching: Even when used purely for two-view matching, the system outperformed transformer-based models like LoFTR on indoor scenes, proving the efficacy of its iterative refinement in low-texture environments.

Conclusion

By coupling deep learning with differentiable geometric solvers, this work provides a robust framework for multi-session SLAM. While challenges remain regarding absolute monocular scale and computational scaling for very large environments, the unified architecture demonstrates a significant leap in handling disjoint video sequences and extreme camera motions.

Summary updated

BibTeX
@inproceedings{lipson2024multi,
  author = {Lahav Lipson and Jia Deng},
  title = {{Multi-Session SLAM with Differentiable Wide-Baseline Pose Optimization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01856},
}
Figure