TL;DR
EMoDepth is a new self-supervised framework that trains a neural network to estimate depth from sparse event camera data by using standard intensity frames as a "teacher" during training. By leveraging cross-modal consistency, the model achieves state-of-the-art accuracy on benchmark datasets while maintaining an inference speed of over 190 FPS, requiring only event data at test time.
What Problem Does This Paper Address?
Event cameras are bio-inspired sensors that only report per-pixel brightness changes (events) rather than full images. They offer massive advantages for robotics: high dynamic range (no blowing out in sunlight), microsecond latency, and no motion blur. However, teaching them to "see" depth is difficult.
Supervised methods require expensive ground-truth depth maps (from LiDAR), which are hard to collect. Existing self-supervised methods usually try to find "photoconsistency" between consecutive event windows. The authors point out a major flaw here: event data is incredibly sparse (often <15% of pixels) and the "appearance" of events changes wildly based on the direction of motion, making traditional consistency checks unreliable.
Methods
The authors propose a framework called EMoDepth that consists of two main components: Depth-Net and Pose-Net.
- Cross-modal Consistency: During training, the system takes both events and standard intensity frames (which are often captured by the same hardware). The Depth-Net predicts depth from events, and the Pose-Net predicts camera motion from the frames. The key "aha" moment is using these to warp one intensity frame to the next. If the predicted depth and pose are correct, the warped frame should match the actual frame. This allows the dense, rich information in standard images to supervise the sparse event-based network.
- Multi-scale Skip-connections: Standard U-Net architectures use skip-connections to pass information from the encoder to the decoder. Because event data is so sparse, information is easily lost during downsampling. EMoDepth uses a more aggressive multi-scale skip-connection architecture (inspired by UNet3+) that fuses features from multiple encoder levels into each decoder node to better preserve spatial details.
- Event Representation: Events are converted into a spatiotemporal voxel grid (a 3D tensor) to make them compatible with standard Convolutional Neural Networks (CNNs).
Crucially, at inference time, the intensity frames and Pose-Net are discarded. The Depth-Net operates solely on event data.
Datasets
The researchers evaluated their method on two major benchmarks:
- MVSEC (Multi-Vehicle Stereo Event Camera): The standard dataset for event-based depth, featuring indoor and outdoor drone/car sequences.
- DSEC: A newer, much higher-resolution (VGA) dataset for driving scenarios with challenging lighting and faster motions.
Key Insights and Ideas
- Events are poor self-supervisors: The paper demonstrates that trying to match events to events (temporal consistency) leads to high error because the signal is too sparse and polarity-dependent.
- Frames as "Scaffolding": You don't need standard images at runtime to benefit from them. By using them as a supervisory signal during training, the network learns the underlying geometry of the world, which it can then infer from events alone later.
- Efficiency over Recurrence: Many state-of-the-art event depth methods use Recurrent Neural Networks (like GRUs) to track state over time. EMoDepth shows that a well-architected CNN with smart skip-connections can outperform these complex models while being significantly faster.
Key Results
- Accuracy: On the MVSEC dataset, EMoDepth outperformed existing unsupervised event-based methods and even beat some supervised hybrid methods (like RAMNet) in outdoor day and night sequences. For example, in the
outdoor_day1sequence at a 30m cutoff, EMoDepth achieved an average absolute error of 2.65m, compared to 3.13m for the supervised E2Depth. - Speed: EMoDepth is highly efficient. On an NVIDIA GTX 1080Ti, it takes only 5.25ms per frame. This is roughly 5x faster than competing methods like E2Depth (24.22ms) and RAM-Net (25.68ms).
- Robustness: The qualitative results show that EMoDepth produces much cleaner depth maps for distant objects (like trees and sky) where previous methods often produced noisy or "blurry" artifacts.
Limitations and Future Work
- Static World Assumption: Like most self-supervised depth methods, the cross-modal consistency loss assumes the scene is static and the only motion comes from the camera. Moving objects (like other cars) can still confuse the depth estimation.
- Data Representation: While voxel grids work well, they are memory-intensive. The authors suggest that exploring more efficient ways to retain spatiotemporal information without increasing data size is a promising future direction.
- Dynamic Objects: Future work will likely focus on explicitly modeling or removing moving objects from the training signal to improve performance in dense urban traffic.
Summary updated