Abstract
Bundle adjustment (BA) is the common way to solve localization and mapping. It is an iterative process in which a system of non-linear equations is solved using two optimization methods, weighted by a damping factor. In the classic approach, the latter is chosen heuristically by the Levenberg-Marquardt algorithm on each iteration. This might take many iterations, making the process computationally expensive, which might be harmful to real-time applications. We propose to replace this heuristic by viewing the problem in a holistic manner, as a game, and formulating it as a reinforcement-learning task. We set an environment which solves the non-linear equations and train an agent to choose the damping factor in a learned manner. We demonstrate that our approach considerably reduces the number of iterations required to reach the bundle adjustment's convergence, on both synthetic and real-life scenarios. We show that this reduction benefits the classic approach and can be integrated with other bundle adjustment acceleration methods.
Introduction
Simultaneous Localization and Mapping (SLAM) is a foundational technology for autonomous systems, from AR headsets to self-driving vehicles. At its core lies Bundle Adjustment (BA), a heavy-duty optimization process that often accounts for 60%–80% of total execution time. BA iteratively minimizes the reprojection error—the difference between a 3D point's estimated position and its observed position in camera frames.
Standard BA implementations rely on the Levenberg-Marquardt (LM) algorithm, which balances slow-but-stable Gradient Descent with fast-but-risky Gauss-Newton steps. This balance is governed by a damping factor, $\lambda$. Classically, $\lambda$ is adjusted via a simple heuristic: decrease it if the error drops, and increase it if the error rises. This "greedy" approach often results in an excessive number of iterations, creating a significant bottleneck for real-time performance.
Methodology: Optimization as a Strategic Game
We propose a paradigm shift by treating the optimization process as a strategic game played by a Reinforcement Learning (RL) agent. Instead of following a hard-coded heuristic, we train a Soft Actor-Critic (SAC) model to dynamically select the optimal $\lambda$ for each iteration.
The RL framework is defined as follows:
- State ($s_t$): A vector representing the five most recent estimation errors, providing the agent with context on the convergence trajectory.
- Action ($a_t$): The agent outputs a continuous value for the damping factor $\lambda$ to be used in the next BA iteration.
- Reward ($r_t$): To prioritize speed, the agent receives a negative reward proportional to the time spent on each iteration. Reaching convergence triggers a large positive bonus.
Unlike previous attempts that focused on minimizing error in a single step, this holistic approach aims for the shortest path to the final solution across the entire optimization sequence.
Datasets and Generalization
The model was evaluated using two industry-standard benchmarks: KITTI (outdoor driving scenes) and BAL (Bundle Adjustment in the Large, featuring massive point clouds).
A key finding was the effectiveness of training on a Synthetic Random Points Dataset. An agent trained on small, simulated 3D scenes generalized remarkably well to complex, real-world cityscapes. This suggests that the underlying "logic" of convergence is universal across geometric structures, allowing for efficient training without massive real-world datasets.
Results
Our RL-based approach demonstrates significant improvements over traditional solvers. On the KITTI benchmark, the agent reduced the number of iterations required for convergence by a factor of five.
Key performance metrics include:
- 3x Wall-Clock Speedup: On KITTI, total execution time was reduced by 3x, even when accounting for the neural network's inference overhead.
- Strategic Efficiency: The holistic RL approach required 50% fewer iterations than "greedy" learning models that only optimize for immediate error reduction.
- Multiplicative Benefits: Since our method reduces the number of iterations, it is fully compatible with other acceleration techniques that reduce the duration of individual iterations (e.g., faster Hessian inversions).
Conclusion and Future Work
By framing Bundle Adjustment as a reinforcement learning task, we have successfully replaced rigid heuristics with a strategic agent capable of navigating complex optimization landscapes. While the neural network overhead makes this approach less suitable for extremely small problems, the benefits for large-scale SLAM and 3D reconstruction are substantial. Future research will focus on reducing inference latency and extending this "game" framework to related problems like Structure from Motion (SfM).
Summary updated