DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions

Yi-Fan Zuo, Jiaqi Yang, Jiaben Chen, Xia Wang, Yifu Wang, Laurent Kneip

TL;DR

DEVO (Depth-Event Camera Visual Odometry) is a robust tracking framework that swaps the standard RGB camera in an RGB-D setup for a high-resolution event camera. By combining the high dynamic range and temporal resolution of events with sparse depth data, DEVO maintains accurate 6-DoF tracking in scenarios where traditional cameras fail, such as near-total darkness or high-speed motion.

What Problem Does This Paper Address?

Visual Odometry (VO)—the process of estimating a camera's motion through an environment—is a cornerstone of robotics. While RGB-D cameras (like the Microsoft Kinect) are popular for indoor mapping, they have two "Achilles' heels":

  1. Motion Blur: Fast movements smear intensity images, making feature tracking impossible.
  2. Lighting Extremes: Low light or high-contrast scenes (HDR) lead to underexposed or blown-out images.

Event cameras (Dynamic Vision Sensors) solve these issues by reporting pixel-level intensity changes asynchronously with microsecond latency. However, using a single event camera for full 6-DoF motion estimation is mathematically complex and computationally expensive. DEVO addresses this by using a hybrid approach: using the event camera for high-speed tracking and a depth sensor to provide the necessary spatial scale.

Methods

The DEVO pipeline consists of two parallel threads: Mapping and Tracking.

  • Event Representation (Time-Surface Maps): Instead of processing individual events, DEVO aggregates them into Time-Surface Maps (TSM). A TSM is an image where pixel values represent the "recency" of an event using an exponential decay function. This effectively creates a map of moving edges.
  • Mapping Module: This thread runs at a lower frame rate. it takes depth readings from the depth sensor and "warps" them into the event camera's perspective using extrinsic calibration. It extracts a semi-dense depth map by only assigning depth to pixels where the TSM indicates strong edge activity.
  • Tracking Module: This is the high-speed core of the system (running at 100Hz). It performs 3D-2D edge alignment. It takes the semi-dense 3D point cloud from the mapping module and aligns it with the current TSM by minimizing an energy function. This is solved efficiently using a forward compositional Lucas-Kanade method, which keeps the computational load low enough for CPU-only execution.

Datasets

The authors evaluated DEVO on both public and custom datasets:

  • MVSEC (Multi-Vehicle Stereo-Event-Camera): A standard public benchmark featuring indoor drone flights and outdoor driving.
  • Self-Collected Datasets: The authors built a custom hardware rig (Azure Kinect + Prophesee Gen3 event camera + PointGrey industrial camera). They recorded three scenes—cali (calibration board), table (desktop), and sofa (living room)—under various speeds (fast, mid, slow) and lighting conditions (bright, darkish, dim, dark, HDR).

Key Insights and Ideas

  • Events as "Natural" Edge Detectors: Traditional VO often spends significant CPU cycles running Canny edge filters on RGB images. Event cameras naturally produce edge-like data, allowing DEVO to skip the expensive image processing step.
  • Decoupling Depth and Tracking: A major insight is that the system doesn't need a high-frequency depth stream. Because the event-based tracking is so fast and accurate, the depth map (the "map") only needs to be updated occasionally.
  • Robustness to Noise: Unlike previous event-based stereo methods (like ESVO) that try to estimate depth from two event cameras, DEVO uses a direct depth sensor. This makes it significantly more resilient to the "noise" generated by event cameras in high-texture or vibrating environments.

Key Results

  • Superiority in the Dark: In "dark" and "HDR" sequences, DEVO maintained continuous tracking while Canny-VO (an RGB-D baseline) failed completely due to a lack of visible features.
  • Extreme Efficiency: DEVO can maintain stable tracking even when the depth camera's frame rate is throttled down to 1Hz. This suggests massive potential for power-saving in mobile robotics.
  • Competitive Accuracy: On the MVSEC dataset, DEVO outperformed the state-of-the-art event-based stereo method (ESVO) in almost all categories, particularly in fast-flight scenarios where ESVO’s mapping often failed.
  • Hardware Requirements: While competitors like KinectFusion require high-end GPUs, DEVO runs efficiently on a standard 8-core CPU.

Limitations and Future Work

  • Sensor Availability: The primary limitation is the current cost and niche status of high-resolution event cameras compared to mass-produced RGB sensors.
  • Localization Focus: The paper focuses on local odometry (tracking motion relative to a starting point). It does not yet implement "Loop Closure" (recognizing a previously visited place to correct long-term drift), which would be required for a full SLAM (Simultaneous Localization and Mapping) system.
  • Dynamic Objects: The current framework assumes a static environment. Moving objects in the scene could potentially confuse the edge-alignment process.

Summary updated

Abstract

We present a novel real-time visual odometry framework for a stereo setup of a depth and high-resolution event camera. Our framework balances accuracy and robustness against computational efficiency towards strong performance in challenging scenarios. We extend conventional edge-based semi-dense visual odometry towards time-surface maps obtained from event streams. Semi-dense depth maps are generated by warping the corresponding depth values of the extrinsically calibrated depth camera. The tracking module updates the camera pose through efficient, geometric semi-dense 3D-2D edge alignment. Our approach is validated on both public and self-collected datasets captured under various conditions. We show that the proposed method performs comparable to state-of-the-art RGB-D camera-based alternatives in regular conditions, and eventually outperforms in challenging conditions such as high dynamics or low illumination.
BibTeX
@inproceedings{zuo2022devo,
  author = {Yi-Fan Zuo and Jiaqi Yang and Jiaben Chen and Xia Wang and Yifu Wang and Laurent Kneip},
  title = {{DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=HrQ1FxDyWb},
}

Loading related content...

Figure