Toward Efficient and Robust Multiple Camera Visual-inertial Odometry

Yao He, Huai Yu, Wen Yang, Sebastian Scherer

Abstract

Efficiency and robustness are the essential criteria for the visual-inertial odometry (VIO) system. To process massive visual data, the high cost on CPU resources and computation latency limits VIO's possibility in integration with other applications. Recently, the powerful embedded GPUs have great potentials to improve the front-end image processing capability. Meanwhile, multi-camera systems can increase the visual constraints for back-end optimization. Inspired by these insights, we incorporate the GPU-enhanced algorithms in the field of VIO and thus propose a new front-end with NVIDIA Vision Programming Interface (VPI). This new front-end then enables multi-camera VIO feature association and provides more stable back-end pose optimization. Experiments with our new front-end on monocular datasets show the CPU resource occupation rate and computational latency are reduced by 40.4% and 50.6% without losing accuracy compared with the original VIO. The multi-camera system shows a higher VIO initialization success rate and better robustness overall state estimation.

Introduction

Visual-Inertial Odometry (VIO) serves as the "inner ear and eyes" for autonomous robots, enabling precise state estimation by fusing high-frequency visual data with inertial measurements. However, modern state-of-the-art (SOTA) VIO systems face a significant "resource hog" problem. Processing high-resolution video streams is computationally expensive, often consuming 150%–250% of CPU resources on embedded platforms like the NVIDIA Jetson. This leaves minimal overhead for other critical autonomous tasks such as path planning, obstacle avoidance, or object detection.

Furthermore, most VIO systems rely on a single or stereo camera setup with a limited Field of View (FoV). In complex environments, such as narrow corridors or areas with featureless surfaces, these systems are prone to failure if the primary camera becomes occluded or faces a textureless wall. This paper addresses these challenges by offloading front-end processing to the GPU and expanding the visual field through a multi-camera configuration.

System Architecture and Methodology

The proposed system builds upon the VINS-Mono framework, introducing two primary innovations: a GPU-accelerated front-end and a robust multi-camera back-end.

VPI-Enhanced Front-end

To alleviate the CPU bottleneck, the researchers offloaded feature detection (Harris corners) and feature tracking (Pyramidal Lucas-Kanade optical flow) to the GPU using the NVIDIA Vision Programming Interface (VPI). Unlike standard CUDA implementations, VPI provides a highly optimized path for computer vision tasks on NVIDIA hardware, minimizing expensive data transfers between the CPU and GPU.

System architecture of the proposed multi-camera VIO system with VPI-enhanced front-end.

To ensure stable tracking, a quad-tree feature filtering algorithm was implemented. This prevents features from clustering in high-texture regions, ensuring a uniform distribution across the image, which is vital for accurate motion estimation.

Tightly-Coupled Multi-Camera Back-end

The optimization back-end was extended to handle multiple cameras with non-overlapping FoVs. The system intelligently selects a "principal camera" for initialization based on feature stability and parallax. Once initialized, all cameras contribute constraints to a single global bundle adjustment, providing a massive "safety net." If one camera sees a blank wall, the others likely maintain visibility of feature-rich areas, preventing drift.

Experimental Results

The system was evaluated using the standard EuRoC MAV dataset for accuracy and efficiency, alongside a custom pedestrian dataset featuring two non-overlapping fisheye cameras.

Efficiency Gains

On the EuRoC dataset, the VPI-enhanced front-end demonstrated remarkable efficiency. Compared to the original VINS-Mono, the system reduced CPU occupation by 40.4% and computational latency by 50.6%. Crucially, these gains were achieved without sacrificing accuracy; the Root Mean Square Error (RMSE) for translation and rotation remained comparable to or better than the baseline.

Robustness and Accuracy

The multi-camera setup proved significantly more robust in challenging environments. In a "Corridor" test sequence, where monocular systems often fail due to lack of features, the proposed system achieved a 98.57% initialization success rate, compared to just 25.71% for the monocular version.

Comparison of state estimation between VINS-Mono (single camera) and the proposed VINS-VPI-Harris (multiple cameras).

As shown in the trajectory comparisons, the multi-camera system maintains a much more stable and accurate path, especially during complex maneuvers where a single camera might lose track of the environment.

Conclusion and Future Work

By leveraging GPU acceleration via VPI and a multi-camera configuration, this research demonstrates that high-performance VIO can be both efficient and robust on embedded hardware. Freeing up CPU resources allows VIO to coexist with other AI-driven tasks, a critical requirement for fully autonomous systems.

Future work will focus on further optimizing the data pipeline using GStreamer to stream images directly to the GPU. Additionally, the team aims to develop "cross-camera" loop closure, where a location seen by one camera can be recognized later by another, further improving the long-term consistency of the pose graph.

Summary updated

Abstract

Efficiency and robustness are the essential criteria for the visual-inertial odometry (VIO) system. To process massive visual data, the high cost on CPU resources and computation latency limits VIO's possibility in integration with other applications. Recently, the powerful embedded GPUs have great potentials to improve the front-end image processing capability. Meanwhile, multi-camera systems can increase the visual constraints for back-end optimization. Inspired by these insights, we incorporate the GPU-enhanced algorithms in the field of VIO and thus propose a new front-end with NVIDIA Vision Programming Interface (VPI). This new front-end then enables multi-camera VIO feature association and provides more stable back-end pose optimization. Experiments with our new front-end on monocular datasets show the CPU resource occupation rate and computational latency are reduced by 40.4% and 50.6% without losing accuracy compared with the original VIO. The multi-camera system shows a higher VIO initialization success rate and better robustness overall state estimation.
BibTeX
@misc{he2021toward,
  author = {Yao He and Huai Yu and Wen Yang and Sebastian Scherer},
  title = {{Toward Efficient and Robust Multiple Camera Visual-inertial Odometry}},
  year = {2021},
  eprint = {2109.12030},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.12030},
}
Figure