DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions

Yi-Fan Zuo, Jiaqi Yang, Jiaben Chen, Xia Wang, Yifu Wang, Laurent Kneip

Abstract

We present a novel real-time visual odometry framework for a stereo setup of a depth and high-resolution event camera. Our framework balances accuracy and robustness against computational efficiency towards strong performance in challenging scenarios. We extend conventional edge-based semi-dense visual odometry towards time-surface maps obtained from event streams. Semi-dense depth maps are generated by warping the corresponding depth values of the extrinsically calibrated depth camera. The tracking module updates the camera pose through efficient, geometric semi-dense 3D-2D edge alignment. Our approach is validated on both public and self-collected datasets captured under various conditions. We show that the proposed method performs comparable to state-of-the-art RGB-D camera-based alternatives in regular conditions, and eventually outperforms in challenging conditions such as high dynamics or low illumination.

Introduction

Visual Odometry (VO)—the process of estimating a camera's motion through an environment—is a cornerstone of modern robotics and autonomous systems. While RGB-D cameras, such as the Microsoft Kinect, are widely used for indoor mapping, they suffer from two primary limitations: motion blur during fast movements and failure in lighting extremes (low light or high-contrast scenes).

DEVO (Depth-Event Camera Visual Odometry) addresses these challenges by replacing the standard RGB camera with a high-resolution event camera. Event cameras, or Dynamic Vision Sensors, report pixel-level intensity changes asynchronously with microsecond latency, offering high dynamic range and temporal resolution. By combining these sensors with a depth camera to provide spatial scale, DEVO maintains accurate 6-DoF tracking in scenarios where traditional intensity-based cameras fail.

Methodology

The DEVO pipeline is designed for real-time performance on standard hardware, consisting of two parallel threads: Mapping and Tracking.

Event Representation and Mapping

Rather than processing individual events, DEVO aggregates them into Time-Surface Maps (TSM). A TSM is an image where pixel values represent the recency of an event using an exponential decay function, effectively creating a map of moving edges. The Mapping thread runs at a lower frequency, taking depth readings and "warping" them into the event camera's perspective via extrinsic calibration. This results in a semi-dense depth map, where depth is only assigned to pixels with strong edge activity.

Semi-dense depth map generated by DEVO's mapping module for the sofa scene.

Tracking Module

The high-speed tracking core (running at 100Hz) performs 3D-2D edge alignment. It aligns the semi-dense 3D point cloud from the mapping module with the current TSM by minimizing a geometric energy function. This is solved efficiently using a forward compositional Lucas-Kanade method, allowing for stable tracking on a standard CPU without requiring a GPU.

Experimental Setup

The framework was validated on the public MVSEC (Multi-Vehicle Stereo-Event-Camera) benchmark and a suite of self-collected datasets. The authors utilized a custom hardware rig featuring an Azure Kinect, a Prophesee Gen3 event camera, and a PointGrey industrial camera. Testing environments included various scenes—such as a desktop (table) and a living room (sofa)—captured under diverse speeds and lighting conditions ranging from bright to near-total darkness.

The 'sofa' scene from the authors' self-collected dataset used for validation.

Key Results and Discussion

  • Robustness in Extremes: In "dark" and high-dynamic-range (HDR) sequences, DEVO maintained continuous tracking while traditional Canny-VO baselines failed completely due to a lack of visible features in intensity images.
  • Computational Efficiency: DEVO demonstrates extreme efficiency, maintaining stable tracking even when the depth camera's frame rate is throttled to as low as 1Hz. This decoupling of depth and tracking suggests significant potential for power-saving in mobile robotics.
  • Competitive Performance: On the MVSEC dataset, DEVO outperformed state-of-the-art event-based stereo methods (like ESVO), particularly in high-speed flight scenarios where mapping often fails for competitors.
  • Natural Edge Detection: By utilizing event cameras, which naturally respond to edges, DEVO bypasses the expensive CPU cycles typically required for intensity-based edge filtering (e.g., Canny filters).

Conclusion and Future Work

DEVO provides a robust alternative to traditional RGB-D VO, particularly for robots operating in unpredictable or poorly lit environments. While currently focused on local odometry, future work includes the implementation of loop closure for full SLAM capabilities and improving robustness against dynamic objects in the scene. The primary hurdle remains the current niche status and cost of high-resolution event sensors compared to mass-market RGB alternatives.

Summary updated

Abstract

We present a novel real-time visual odometry framework for a stereo setup of a depth and high-resolution event camera. Our framework balances accuracy and robustness against computational efficiency towards strong performance in challenging scenarios. We extend conventional edge-based semi-dense visual odometry towards time-surface maps obtained from event streams. Semi-dense depth maps are generated by warping the corresponding depth values of the extrinsically calibrated depth camera. The tracking module updates the camera pose through efficient, geometric semi-dense 3D-2D edge alignment. Our approach is validated on both public and self-collected datasets captured under various conditions. We show that the proposed method performs comparable to state-of-the-art RGB-D camera-based alternatives in regular conditions, and eventually outperforms in challenging conditions such as high dynamics or low illumination.
BibTeX
@inproceedings{zuo2022devo,
  author = {Yi-Fan Zuo and Jiaqi Yang and Jiaben Chen and Xia Wang and Yifu Wang and Laurent Kneip},
  title = {{DEVO: Depth-Event Camera Visual Odometry in Challenging Conditions}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=HrQ1FxDyWb},
}
Figure