Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios

Antoni Rosinol Vidal, Henri Rebecq, Timo Horstschaefer, Davide Scaramuzza

TL;DR

This paper presents the first visual SLAM pipeline that tightly fuses event cameras, standard intensity frames, and inertial measurements (IMU) to achieve robust state estimation in both high-speed, high-dynamic-range (HDR) environments and static scenes. The authors leverage this hybrid approach to demonstrate the first autonomous quadrotor flight using an event camera for state estimation, unlocking stable flight through extreme conditions like mid-air light-switching and high-speed maneuvers.

What Problem Does This Paper Address?

Robots navigating autonomously rely heavily on Visual-Inertial Odometry (VIO) to estimate their position and orientation. However, traditional VIO pipelines use standard cameras, which fail in two common edge cases:

  1. High-speed motion, which causes severe motion blur due to fixed frame-rate exposure times.
  2. High Dynamic Range (HDR) scenes (e.g., moving from a dark room into bright sunlight), which lead to severely under- or over-exposed images.

Bio-inspired event cameras solve these issues by reporting asynchronous, pixel-level brightness changes instead of full frames. They boast microsecond latency, do not suffer from motion blur, and operate at an outstanding 140 dB dynamic range. However, event cameras have a major Achilles' heel: when the robot stops moving, the camera stops generating data.

To build a truly robust system, the authors address the challenge of combining the complementary strengths of standard cameras (rich information when slow/stationary) and event cameras (reliable data when fast/HDR) alongside an IMU.

Methods

The proposed pipeline, termed Ultimate SLAM, processes data from a hybrid sensor setup containing an event camera, a standard camera, and an IMU.

Our state estimation pipeline combines events, standard frames, and inertial measurements to provide robust state estimation.

Key algorithmic steps include:

  • Spatio-temporal Event Windows: Events are grouped into windows synchronized directly to the timestamps of incoming standard frames.
  • Motion-Compensated Event Frames: Using high-frequency IMU data and estimated scene depth, the asynchronous events within a window are mathematically "un-distorted" to synthesize sharp, motion-compensated virtual event frames.
  • Parallel Feature Tracking: Visual features (FAST corners) are detected and tracked independently using a Lucas-Kanade (KLT) tracker on both the standard frames and the synthesized event frames.
  • Tightly-Coupled Nonlinear Optimization: The feature tracks from both heterogeneous sources, along with pre-integrated IMU data, are fed into a joint cost function. A sliding-window optimization (using Google Ceres) minimizes the visual reprojection errors and inertial errors simultaneously.
  • No-Motion Prior: To prevent drift when the robot is stationary (and the event camera goes blind), the pipeline automatically injects a strong zero-velocity constraint into the optimization whenever the event rate drops below a specific threshold.

Datasets

The pipeline was quantitatively evaluated using the publicly available Event Camera Dataset, which contains aggressive quadrotor trajectories and HDR scenes recorded with a DAVIS240C sensor. This hardware sensor natively outputs synchronized 24Hz standard frames, asynchronous events, and 1kHz IMU measurements.

Key Insights and Ideas

  • Natural Modality Switching: By formulating the fusion as a joint optimization problem, the system avoids brittle, hard-coded switching logic. When standard frames degrade due to motion blur or poor lighting, their residual errors increase, and the optimization naturally relies more heavily on the clean feature tracks provided by the event camera.
  • Overcoming Stillness: The addition of an event-rate-triggered "no-motion prior" elegantly solves the long-standing issue of event-based SLAM drifting during periods of inactivity or hovering.

Key Results

The hybrid pipeline significantly outperforms single-modality baselines on the Event Camera Dataset:

  • 130% accuracy improvement over event-only pipelines (Events + IMU).
  • 85% accuracy improvement over traditional visual-inertial systems (Standard Frames + IMU).
Comparison of the proposed approach against single-modality baselines across different segments of the trajectory.

Autonomous Flight Demonstration

The authors successfully implemented the pipeline to run in real-time onboard an autonomous quadrotor equipped with an Odroid XU4 computer. In flight experiments, the quadrotor successfully maintained stable closed-loop control under extreme conditions:

  • The "Light Switch" Test: The drone safely flew in autonomous circles even when the room lights were abruptly turned completely off and on, a scenario where standard VIO completely fails.
  • High-Speed Circles: The drone executed aggressive circular trajectories generating an optical flow of 340 pixels/second, maintaining accurate tracking despite severe motion blur on the standard camera frames.

Limitations and Future Work

While highly robust, the authors note a few limitations:

  • Global Illumination Bursts: Abruptly flipping a light switch causes nearly every pixel on an event camera to fire simultaneously. While the pipeline recovers quickly by re-initializing features, it does not explicitly model or filter out these global illumination artifacts.
  • Feature-Level Fusion: The current architecture tracks features independently on standard and event frames before fusing them in the optimizer. Future work could explore tighter pixel-level fusion to directly guide feature tracking on standard frames using raw event streams.

Summary updated

Abstract

Event cameras are bio-inspired vision sensors that output pixel-level brightness changes instead of standard intensity frames. These cameras do not suffer from motion blur and have a very high dynamic range, which enables them to provide reliable visual information during high speed motions or in scenes characterized by high dynamic range. However, event cameras output only little information when the amount of motion is limited, such as in the case of almost still motion. Conversely, standard cameras provide instant and rich information about the environment most of the time (in low-speed and good lighting scenarios), but they fail severely in case of fast motions, or difficult lighting such as high dynamic range or low light scenes. In this paper, we present the first state estimation pipeline that leverages the complementary advantages of these two sensors by fusing in a tightly-coupled manner events, standard frames, and inertial measurements. We show on the publicly available Event Camera Dataset that our hybrid pipeline leads to an accuracy improvement of 130% over event-only pipelines, and 85% over standard-frames-only visual-inertial systems, while still being computationally tractable. Furthermore, we use our pipeline to demonstrate - to the best of our knowledge - the first autonomous quadrotor flight using an event camera for state estimation, unlocking flight scenarios that were not reachable with traditional visual-inertial odometry, such as low-light environments and high-dynamic range scenes.
BibTeX
@article{vidal2018ultimate,
  author = {Antoni Rosinol Vidal and Henri Rebecq and Timo Horstschaefer and Davide Scaramuzza},
  title = {{Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2018},
  doi = {10.1109/lra.2018.2793357},
}
Figure