Abstract
This work presents a novel, self-supervised framework for real-time estimation of object height and depth using event cameras on mobile platforms. By leveraging the geometric principles of "Plane and Parallax" (P+P), the proposed system achieves high-frequency (75Hz) performance and demonstrates superior robustness in challenging environments, such as high-speed driving and low-light night scenarios, where traditional frame-based cameras typically fail.
Introduction
Reliable 3D environmental perception is a cornerstone of autonomous navigation. However, conventional RGB cameras are often limited by motion blur at high velocities and restricted dynamic range in nocturnal settings. Event cameras, which asynchronously report pixel-level intensity changes, offer a hardware-level solution to these issues. Despite their advantages, interpreting sparse event data requires specialized algorithms.
Existing deep learning models for depth estimation frequently struggle with generalization, often overfitting to specific scene semantics rather than learning fundamental geometry. Moreover, dense optical flow computation is computationally prohibitive for real-time applications. This paper addresses these challenges by introducing a computationally efficient, geometry-aware method for scene structure estimation.
Methodology
The core of the approach is the Plane and Parallax (P+P) geometric framework, which simplifies the depth estimation problem by isolating off-plane motion.
- Event Representation: Raw event streams are converted into a 3D spatiotemporal volume using trilinear interpolation, preserving critical timing information.
- Geometric Warping: Assuming a dominant ground plane, the system applies a homography based on camera pose to "warp" the event data. This process effectively cancels out the optical flow for any points residing on the ground plane.
- Residual Parallax: Objects extending above the ground plane exhibit "residual flow" in the warped space. This residual motion is geometrically linked to the object's height.
- Network Architecture: A convolutional neural network (CNN) is trained to predict the ratio between height ($h$) and depth ($\rho$). Predicting this ratio, rather than absolute values, enhances the model's ability to generalize across diverse environments.
- Self-Supervised Learning: The model is trained without LiDAR ground truth. Instead, it utilizes a photometric loss, minimizing the discrepancy between successive camera images after warping them according to the predicted scene structure.
Experimental Setup
The system was evaluated using the Multi Vehicle Stereo Event Camera (MVSEC) dataset. Training was conducted on the outdoor_day2 suburban sequence. To test generalization, evaluation was performed on outdoor_day1 (office park) and multiple outdoor_night sequences. Additionally, high-speed robustness was assessed using a motorcycle sequence reaching speeds of 140 km/hr.
Results and Discussion
The "Event P+P" method demonstrates significant improvements over baseline approaches, particularly in low-light conditions.
Key Performance Metrics
- Real-Time Efficiency: The pipeline operates at 75Hz on modern GPU hardware, meeting the requirements for high-speed autonomous systems.
- Nocturnal Superiority: In the
outdoor_night1sequence, Event P+P achieved an average depth error of 2.93m, vastly outperforming the standard Event Depth baseline of 15.62m. This highlights the advantage of event cameras' high dynamic range. - Freespace Detection: By thresholding the predicted heights, the system effectively identifies drivable "free space," providing a practical tool for obstacle avoidance.
- High-Speed Robustness: Qualitative analysis of motorcycle data confirmed the system's ability to accurately detect vehicles at highway speeds where traditional cameras suffer from severe motion blur.
Conclusion
By integrating classical geometric constraints with deep learning, this work provides a robust solution for real-time 3D perception. While the method relies on a dominant ground plane assumption and precise camera calibration, it offers a significant leap in performance for autonomous vehicles operating in high-speed or low-light environments. Future research will focus on relaxing the ground plane assumption for off-road navigation and improving the handling of highly dynamic objects.
Summary updated