ESVIO: Event-Based Stereo Visual-Inertial Odometry

Zhe Liu, Dianxi Shi, Ruihao Li, Shaowu Yang

TL;DR

This paper introduces ESVIO, a novel event-based stereo visual-inertial odometry system that leverages the unique capabilities of event cameras for robust and accurate motion estimation. ESVIO addresses key challenges in event-based VIO by proposing a direct, feature-less approach for state estimation and a novel semi-joint initialization method, leading to improved accuracy and robustness in challenging scenarios.

What Problem Does This Paper Address?

Traditional visual odometry (VO) systems struggle in challenging environments characterized by high dynamic range, rapid illumination changes, and fast motion. Event cameras, with their high temporal resolution, low latency, and wide dynamic range, offer a promising alternative. However, existing event-based VO methods often face issues with low spatial resolution, high noise, and difficulty in feature extraction and tracking. Furthermore, a critical challenge for any VIO system is accurate initialization, which involves estimating parameters like scale, gravity direction, and sensor biases. For event-based VIO, effective initialization methods that fuse event data with IMU measurements are lacking. This paper aims to bridge this gap by proposing ESVIO, a system that combines a direct event-based VIO approach with a robust event-inertial initialization strategy.

Methods

ESVIO is built upon two core novel contributions:

  • Direct Event-Based Stereo VIO: Instead of relying on traditional feature extraction and tracking, ESVIO employs a direct method. It fuses event data (specifically, event depth and Time-Surface images) with pre-integrated inertial measurements within a sliding window non-linear optimization framework. This approach directly estimates camera motion and IMU biases, aiming for improved accuracy and robustness, especially in texture-less or rapidly changing environments where feature-based methods might fail.
  • Semi-Joint Event-Inertial Initialization: This method addresses the crucial initialization problem for event-based VIO. It consists of two steps:
  • Event-Only Initialization: This step uses event data to obtain an "up-to-scale" camera trajectory. This provides initial constraints for the subsequent optimization.
  • Event-Inertial Initial Optimization: This step refines the initial estimates by jointly optimizing all initialization parameters (scale, gravity direction, initial velocity, and IMU biases) using the event-based trajectory and IMU measurements within a sliding window. This semi-joint approach aims to achieve both rapid and accurate initialization.

The system also incorporates an improved Time-Surface (TS) image generation method to better handle slow camera motion and a novel inverse depth frame reconstruction technique to filter redundant depth information, thereby optimizing computational efficiency.

Datasets

ESVIO was evaluated on several public datasets:

  • RPG dataset [25]: Collected with a handheld stereo event camera in an indoor environment.
  • MVSEC dataset [27]: Sequences collected using a stereo event camera mounted on a drone in a capacious indoor environment.
  • ESIM dataset [26]: Generated using the event camera simulator ESIM.
  • DSEC dataset [28]: Collected from a stereo event camera mounted on a driving car, featuring higher resolution and challenging driving scenarios.

The paper notes that none of these datasets provide ground truth IMU bias, which limits direct evaluation of the initialization's impact on bias estimation.

Key Insights and Ideas

  • Direct VIO for Event Cameras is Viable: The paper demonstrates that a direct, feature-less approach can be effective for event-based VIO, overcoming some limitations of feature-based methods in challenging scenarios.
  • Initialization is Crucial and Achievable for Event-VIO: The proposed semi-joint initialization method significantly improves the performance of event-based VIO, particularly in scale recovery and overall state estimation accuracy.
  • Improved Time-Surface for Slow Motion: The modified TS generation method effectively addresses the issue of insufficient intensity information in slow-motion scenarios, which is a limitation of standard TS approaches.
  • Stereo Event Cameras Offer Robustness: The use of stereo event cameras, combined with IMU data, leads to more accurate and robust state estimation compared to monocular approaches or event-only VO.

Key Results

  • Accuracy Improvement: ESVIO consistently outperforms state-of-the-art event-based stereo VO methods (like ESVO and its variants) in terms of mean Absolute Translational Error (ATE) across multiple datasets. For instance, on the ESIM dataset, ESVIO achieved a mean ATE of 4.5 cm, significantly better than ESVO's 10.4 cm.
  • Initialization Impact: ESVIO with its proposed event-inertial initialization shows significantly lower scale errors at both the beginning and end of sequences compared to a variant without initialization. This highlights the effectiveness of the initialization in recovering scale and improving overall accuracy.
  • Robustness: ESVIO demonstrates better robustness, particularly in scenarios with significant rotations or fast camera motion, as evidenced by lower translational and rotational errors.
  • Real-time Performance: The system maintains real-time performance, with the VIO thread operating at approximately 42 Hz. However, it notes that for very heavy event loads (like in the DSEC dataset), real-time performance might be compromised, requiring slower playback.

Limitations and Future Work

  • Event Camera Hardware Limitations: The performance of ESVIO is still constrained by the inherent limitations of current event camera hardware (e.g., noise, spatial resolution).
  • Lack of Long-Term Data Association: ESVIO does not currently incorporate long-term data association mechanisms like loop closure, which are essential for relocalization and drift elimination in large-scale environments.
  • Future Work: The authors plan to:
  • Integrate standard camera frames to further enhance performance.
  • Implement loop closure for improved relocalization and drift correction.
  • Develop a specific marginalization step for ESVIO.
  • Create a new event camera dataset specifically tailored for event-based VIO and initialization methods.

Summary updated

Abstract

The emerging event cameras are bio-inspired sensors that can output pixel-level brightness changes at extremely high rates, and event-based visual-inertial odometry (VIO) is widely studied and used in autonomous robots. In this paper, we propose an event-based stereo VIO system, namely ESVIO. Firstly, we present a novel direct event-based VIO method, which fuses events’ depth, Time-Surface images, and pre-integrated inertial measurement to estimate the camera motion and inertial measurement unit (IMU) biases in a sliding window non-linear optimization framework, effectively improving the state estimation accuracy and robustness. Secondly, we design an event-inertia semi-joint initialization method, through two steps of event-only initialization and event-inertia initial optimization, to rapidly and accurately solve the initialization parameters of the VIO system, thereby further improving the state estimation accuracy. Based on these two methods, we implement the ESVIO system and evaluate the effectiveness and robustness of ESVIO on various public datasets. The experimental results show that ESVIO achieves good performance in both accuracy and robustness when compared with other state-of-the-art event-based VIO and stereo visual odometry (VO) systems, and, at the same time, with no compromise to real-time performance.
BibTeX
@inproceedings{liu2023esvio,
  author = {Zhe Liu and Dianxi Shi and Ruihao Li and Shaowu Yang},
  title = {{ESVIO: Event-Based Stereo Visual-Inertial Odometry}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=iZhiBRqTvRt},
}
Figure