Stereo Direct Sparse Visual–Inertial Odometry with Efficient Second-Order Minimization

Chenhui Fu, Jiangang LuSensors 2025 · DOI

TL;DR

This paper introduces SDS-VIO, a stereo direct sparse visual-inertial odometry system that tackles key challenges in VIO like initialization sensitivity and dynamic illumination. By employing an efficient second-order minimization (ESM) approach and a novel visual-inertial initialization strategy, SDS-VIO achieves state-of-the-art accuracy and robustness, outperforming existing methods on challenging benchmarks.

What Problem Does This Paper Address?

Autonomous systems, from micro-aerial vehicles to augmented reality devices, heavily rely on accurate ego-motion estimation, especially in GPS-denied environments. Visual-Inertial Odometry (VIO) is a leading technology for this, combining camera and Inertial Measurement Unit (IMU) data. However, current VIO systems face significant hurdles:

  • Initialization Sensitivity: Many systems require specific, often slow, motions to initialize correctly, making them prone to failure with aggressive movements.
  • Dynamic Illumination: Direct methods, which rely on photometric consistency, struggle with changing lighting conditions, a common issue in real-world scenarios.
  • Multi-Sensor Fusion: Effectively integrating data from visual and inertial sensors, which have different temporal characteristics, is complex and can lead to error propagation.

This paper aims to overcome these limitations by proposing a stereo direct sparse VIO system that is more robust, accurate, and less sensitive to initialization procedures.

Methods

The proposed Stereo Direct Sparse Visual-Inertial Odometry (SDS-VIO) system is built upon a direct method approach and incorporates several key innovations:

  • Visual-Inertial Initialization: Instead of relying on specific initialization motions, SDS-VIO uses a two-stage process. First, it performs spatial static stereo matching to estimate initial depth. This is followed by a visual-inertial alignment that integrates IMU preintegration uncertainty, enabling reliable initialization even under arbitrary initial motions.
  • Efficient Second-Order Minimization (ESM): The core optimization uses an ESM algorithm. This approach approximates the photometric error's curvature using a second-order Taylor expansion and the Jacobian with a first-order expansion. This leads to more efficient and accurate optimization compared to traditional methods, especially under challenging photometric conditions.
  • Direct Image Alignment: The system tracks new stereo frames by optimizing photometric and inertial errors directly on raw pixel intensities, avoiding the limitations of feature extraction.
  • Adaptive Keyframe Selection: An adaptive tracking ratio is defined to determine when to add new keyframes. This strategy balances efficiency and robustness by adding keyframes only when necessary, preventing the system from becoming computationally bogged down.
  • Marginalization: To manage computational complexity, a marginalization module is employed. This process converts previous measurements into a prior term, preserving essential information while reducing the size of the optimization problem.

Datasets

The proposed SDS-VIO system was evaluated on two widely recognized datasets:

  • KITTI Visual Odometry Benchmark: This dataset consists of sequences collected from a moving car, featuring street scenes with dynamic objects. The evaluation focused on the first 11 sequences for which ground-truth 6D poses are available.
  • EuRoC Dataset: This dataset comprises sequences collected from Micro Aerial Vehicles (MAVs) in two environments: an industrial machine hall and a Vicon room. It presents challenges such as low illumination, motion blur, and low-texture features.

Key Insights and Ideas

  • Initialization without Restrictive Motions: The ability to initialize reliably using arbitrary initial motions is a significant step towards more practical VIO systems, reducing user burden and increasing deployment flexibility.
  • ESM for Robustness: The adoption of ESM for optimization proves crucial for handling dynamic illumination and geometric transformations, leading to more stable and accurate tracking in challenging real-world conditions.
  • Adaptive Keyframe Management: The adaptive tracking ratio for keyframe selection is a clever way to dynamically adjust the system's complexity, ensuring efficiency without sacrificing performance.
  • Tight Visual-Inertial Coupling: The joint optimization of visual and inertial factors, along with the incorporation of IMU preintegration uncertainty, leads to a more cohesive and accurate state estimation.

Key Results

  • KITTI Benchmark: SDS-VIO demonstrated state-of-the-art performance, outperforming existing methods like R-SDSO and SDSO in terms of average translational and rotational errors. For instance, on sequence 00, SDS-VIO achieved a translational error of 0.74% and rotational error of 0.27 deg/100m, which were better than R-SDSO (0.90, 0.30) and SDSO (1.10, 0.34).
  • EuRoC Dataset: The system showed remarkable robustness, particularly in challenging sequences like V2_03_difficult, where other methods like BASALT and OKVIS failed to track. SDS-VIO achieved the best mean Absolute Trajectory Error (ATE) of 0.061 across all sequences.
  • Speed: SDS-VIO with ESM achieved a fast tracking time of 42.67 ms per frame, outperforming SDSO (79.600 ms) and VINS-Fusion (82.670 ms), indicating its potential for real-time applications.

Limitations and Future Work

  • Limitations: The paper notes that while SDS-VIO performs well, the translational errors on the KITTI dataset showed slight variations, potentially due to the dataset's relatively low frame rate impacting IMU effectiveness. The authors also mention that it's difficult to ensure a completely fair comparison with other systems due to differences in window sizes, pyramid levels, and other hyper-parameters.
  • Future Work: The authors suggest future research directions including developing a database for map maintenance and incorporating loop closure. These additions would further enhance the accuracy of SDS-VIO and extend its capabilities to a full visual-inertial fused SLAM system.

Summary updated

Abstract

Visual-inertial odometry (VIO) is the primary supporting technology for autonomous systems, but it faces three major challenges: initialization sensitivity, dynamic illumination, and multi-sensor fusion. In order to overcome these challenges, this paper proposes stereo direct sparse visual-inertial odometry with efficient second-order minimization. It is entirely implemented using the direct method, which includes a depth initialization module based on visual-inertial alignment, a stereo image tracking module, and a marginalization module. Inertial measurement unit (IMU) data is first aligned with a stereo image to initialize the system effectively. Then, based on the efficient second-order minimization (ESM) algorithm, the photometric error and the inertial error are minimized to jointly optimize camera poses and sparse scene geometry. IMU information is accumulated between several frames using measurement preintegration and is inserted into the optimization as an additional constraint between keyframes. A marginalization module is added to reduce the computation complexity of the optimization and maintain the information about the previous states. The proposed system is evaluated on the KITTI visual odometry benchmark and the EuRoC dataset. The experimental results demonstrate that the proposed system achieves state-of-the-art performance in terms of accuracy and robustness.
BibTeX
@inproceedings{fu2025stereo,
  author = {Chenhui Fu and Jiangang Lu},
  title = {{Stereo Direct Sparse Visual–Inertial Odometry with Efficient Second-Order Minimization}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25154852},
}
Figure