INCREMENTAL REAL-TIME BUNDLE ADJUSTMENT FOR MULTI-CAMERA SYSTEMS WITH POINTS AT INFINITY

Johannes Schneider, Thomas Läbe, Wolfgang Förstner2013 · DOI

Abstract

This paper presents a concept and first experiments on a keyframe-based incremental bundle adjustment for real-time structure and motion estimation in an unknown scene. In order to avoid periodic batch steps, we use the software iSAM2 for sparse nonlinear incremental optimization, which is highly efficient through incremental variable reordering and fluid relinearization. We adapted the software to allow for (1) multi-view cameras by taking the rigid transformation between the cameras into account, (2) omnidirectional cameras as it can handle arbitrary bundles of rays and (3) scene points at infinity, which improve the estimation of the camera orientation as points at the horizon can be observed over long periods of time. The real-time bundle adjustment refers to sets of keyframes, consisting of frames, one per camera, taken in a synchronized way, that are initiated if a minimal geometric distance to the last keyframe set is exceeded. It uses interest points in the keyframes as observations, which are tracked in the synchronized video streams of the individual cameras and matched across the cameras, if possible. First experiments show the potential of the incremental bundle adjustment w.r.t. time requirements. Our experiments are based on a multi-camera system with four fisheye cameras, which are mounted on a UAV as two stereo pairs, one looking ahead and one looking backwards, providing a large field of view.

Introduction

For autonomous Unmanned Aerial Vehicles (UAVs) to navigate unknown environments, precise Visual Odometry (VO) is essential. While Bundle Adjustment (BA) is the gold standard for accuracy—simultaneously refining 3D scene points and camera trajectories—it is traditionally computationally expensive. Standard BA often requires periodic "batch" processing that re-calculates the entire state, leading to latency issues that are problematic for high-speed flight. Furthermore, distant landmarks near the horizon are often discarded as noise in traditional systems because their depth cannot be accurately triangulated, despite their immense value for stabilizing orientation.

The quadrocopter UAV equipped with a multi-camera rig consisting of four fisheye cameras.

Methodology

The proposed system utilizes a custom quadrocopter equipped with four fisheye cameras arranged as two stereo pairs. The software architecture is built upon three core pillars:

  1. Incremental Optimization via iSAM2: Instead of full batch re-optimization, the system employs the iSAM2 algorithm. By using a Bayes Tree data structure, iSAM2 performs "fluid relinearization," updating only the parts of the map and trajectory affected by new observations. This maintains real-time performance even as the mission duration increases.
  2. Omnidirectional Multi-Camera Modeling: The mathematical framework is adapted to handle extreme wide-angle (185°) fisheye lenses and the rigid physical offsets between the four cameras. This allows the system to treat the rig as a single, wide-FOV sensor.
  3. Integration of Points at Infinity: Distant points that lack measurable depth are explicitly tracked. While they do not provide translational data, they serve as a stable "visual compass," significantly reducing rotational drift over time.
Synchronized views from the four fisheye cameras with tracked interest points used for bundle adjustment.

To optimize computational resources, the system utilizes a keyframe-based approach. New optimization cycles are triggered only when the UAV has moved a significant geometric distance (e.g., 1 meter), ensuring that every update contributes meaningful new information to the trajectory estimation.

Experimental Evaluation

The system was tested using a real-world dataset consisting of 1,800 frame sets captured at 14 Hz during circular flight maneuvers. The hardware rig featured four Lensagon fisheye cameras on a 20 cm base. Results were compared against BACS, a rigorous batch bundle adjustment tool, to establish a baseline for accuracy.

Key findings include:

  • Real-Time Performance: Incremental updates consistently stayed under 1 second on a 3.6 GHz machine.
  • Statistical Optimality: The accuracy of the incremental method was found to be nearly identical to the batch method, with deviations remaining within the estimated uncertainty range.
  • Robustness: The implementation of a Huber cost function allowed the system to effectively ignore outliers and erroneous feature matches.

Conclusion and Future Work

While the system demonstrates high potential for real-time UAV navigation, some limitations remain. The researchers observed a Root Mean Square Error (RMSE) of 2–3 pixels, likely caused by high-frequency rotor vibrations affecting the "rigid rig" assumption. Future work will focus on creating a more seamless mathematical transition for points moving from the distance toward the drone and integrating GPS and IMU data into the iSAM2 framework for enhanced absolute positioning and maneuver resilience.

Summary updated

Abstract

This paper presents a concept and first experiments on a keyframe-based incremental bundle adjustment for real-time structure and motion estimation in an unknown scene. In order to avoid periodic batch steps, we use the software iSAM2 for sparse nonlinear incremental optimization, which is highly efficient through incremental variable reordering and fluid relinearization. We adapted the software to allow for (1) multi-view cameras by taking the rigid transformation between the cameras into account, (2) omnidirectional cameras as it can handle arbitrary bundles of rays and (3) scene points at infinity, which improve the estimation of the camera orientation as points at the horizon can be observed over long periods of time. The real-time bundle adjustment refers to sets of keyframes, consisting of frames, one per camera, taken in a synchronized way, that are initiated if a minimal geometric distance to the last keyframe set is exceeded. It uses interest points in the keyframes as observations, which are tracked in the synchronized video streams of the individual cameras and matched across the cameras, if possible. First experiments show the potential of the incremental bundle adjustment w.r.t. time requirements. Our experiments are based on a multi-camera system with four fisheye cameras, which are mounted on a UAV as two stereo pairs, one looking ahead and one looking backwards, providing a large field of view.
BibTeX
@inproceedings{schneider2013incremental,
  author = {Johannes Schneider and Thomas Läbe and Wolfgang Förstner},
  title = {{INCREMENTAL REAL-TIME BUNDLE ADJUSTMENT FOR MULTI-CAMERA SYSTEMS WITH POINTS AT INFINITY}},
  booktitle = {˜The},
  year = {2013},
  doi = {10.5194/isprsarchives-xl-1-w2-355-2013},
}
Figure