Self-supervised Event-based Monocular Depth Estimation using Cross-modal Consistency

Junyu Zhu, Lina Liu, Bofeng Jiang, Feng Wen, Hongbo Zhang, Wanlong Li, Yong Liu

Abstract

An event camera is a novel vision sensor that can capture per-pixel brightness changes and output a stream of asynchronous "events". It has advantages over conventional cameras in those scenes with high-speed motions and challenging lighting conditions because of the high temporal resolution, high dynamic range, low bandwidth, low power consumption, and no motion blur. Therefore, several supervised monocular depth estimation from events is proposed to address scenes difficult for conventional cameras. However, depth annotation is costly and time-consuming. In this paper, to lower the annotation cost, we propose a self-supervised event-based monocular depth estimation framework named EMoDepth. EMoDepth constrains the training process using the cross-modal consistency from intensity frames that are aligned with events in the pixel coordinate. Moreover, in inference, only events are used for monocular depth prediction. Additionally, we design a multi-scale skip-connection architecture to effectively fuse features for depth estimation while maintaining high inference speed. Experiments on MVSEC and DSEC datasets demonstrate that our contributions are effective and that the accuracy can outperform existing supervised event-based and unsupervised frame-based methods.

TL;DR

EMoDepth is a new self-supervised framework that trains a neural network to estimate depth from sparse event camera data by using standard intensity frames as a "teacher" during training. By leveraging cross-modal consistency, the model achieves state-of-the-art accuracy on benchmark datasets while maintaining an inference speed of over 190 FPS, requiring only event data at test time.

The Challenge: Depth from Sparse Events

Event cameras are bio-inspired sensors that only report per-pixel brightness changes (events) rather than full images. They offer massive advantages for robotics: high dynamic range (no blowing out in sunlight), microsecond latency, and no motion blur. However, teaching them to "see" depth is difficult.

Comparison between standard intensity frames (top) and sparse event data (bottom).

Supervised methods require expensive ground-truth depth maps (from LiDAR), which are hard to collect. Existing self-supervised methods usually try to find "photoconsistency" between consecutive event windows. The authors point out a major flaw here: event data is incredibly sparse (often <15% of pixels) and the "appearance" of events changes wildly based on the direction of motion, making traditional consistency checks unreliable.

Methods: EMoDepth Framework

The authors propose a framework called EMoDepth that consists of two main components: Depth-Net and Pose-Net.

The EMoDepth training and testing paradigm. During training, aligned frames provide supervision; at test time, only events are required.
  1. Cross-modal Consistency: During training, the system takes both events and standard intensity frames (which are often captured by the same hardware). The Depth-Net predicts depth from events, and the Pose-Net predicts camera motion from the frames. The key innovation is using these to warp one intensity frame to the next. If the predicted depth and pose are correct, the warped frame should match the actual frame. This allows the dense, rich information in standard images to supervise the sparse event-based network.
  2. Multi-scale Skip-connections: Standard U-Net architectures use skip-connections to pass information from the encoder to the decoder. Because event data is so sparse, information is easily lost during downsampling. EMoDepth uses a more aggressive multi-scale skip-connection architecture (inspired by UNet3+) that fuses features from multiple encoder levels into each decoder node to better preserve spatial details.
  3. Event Representation: Events are converted into a spatiotemporal voxel grid (a 3D tensor) to make them compatible with standard Convolutional Neural Networks (CNNs).

Crucially, at inference time, the intensity frames and Pose-Net are discarded. The Depth-Net operates solely on event data.

Key Results

The researchers evaluated their method on two major benchmarks: MVSEC (Multi-Vehicle Stereo Event Camera) and DSEC (a high-resolution driving dataset).

  • Accuracy: On the MVSEC dataset, EMoDepth outperformed existing unsupervised event-based methods and even beat some supervised hybrid methods (like RAMNet) in outdoor day and night sequences. For example, in the outdoor_day1 sequence at a 30m cutoff, EMoDepth achieved an average absolute error of 2.65m, compared to 3.13m for the supervised E2Depth.
  • Speed: EMoDepth is highly efficient. On an NVIDIA GTX 1080Ti, it takes only 5.25ms per frame. This is roughly 5x faster than competing methods like E2Depth (24.22ms) and RAM-Net (25.68ms).
  • Robustness: Qualitative results show that EMoDepth produces much cleaner depth maps for distant objects (like trees and sky) where previous methods often produced noisy or "blurry" artifacts.

Insights and Future Work

EMoDepth demonstrates that you don't need standard images at runtime to benefit from them. By using them as a "scaffolding" signal during training, the network learns the underlying geometry of the world, which it can then infer from events alone later. Furthermore, the work shows that a well-architected CNN with smart skip-connections can outperform complex recurrent models while being significantly faster.

Future work will likely focus on explicitly modeling or removing moving objects from the training signal to improve performance in dense urban traffic, as the current cross-modal consistency loss assumes a static world.

Summary updated

Abstract

An event camera is a novel vision sensor that can capture per-pixel brightness changes and output a stream of asynchronous ``events''. It has advantages over conventional cameras in those scenes with high-speed motions and challenging lighting conditions because of the high temporal resolution, high dynamic range, low bandwidth, low power consumption, and no motion blur. Therefore, several supervised monocular depth estimation from events is proposed to address scenes difficult for conventional cameras. However, depth annotation is costly and time-consuming. In this paper, to lower the annotation cost, we propose a self-supervised event-based monocular depth estimation framework named EMoDepth. EMoDepth constrains the training process using the cross-modal consistency from intensity frames that are aligned with events in the pixel coordinate. Moreover, in inference, only events are used for monocular depth prediction. Additionally, we design a multi-scale skip-connection architecture to effectively fuse features for depth estimation while maintaining high inference speed. Experiments on MVSEC and DSEC datasets demonstrate that our contributions are effective and that the accuracy can outperform existing supervised event-based and unsupervised frame-based methods.
BibTeX
@inproceedings{zhu2023self,
  author = {Junyu Zhu and Lina Liu and Bofeng Jiang and Feng Wen and Hongbo Zhang and Wanlong Li and Yong Liu},
  title = {{Self-supervised Event-based Monocular Depth Estimation using Cross-modal Consistency}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=qG2RagAswz},
}
Figure