Learning Event-Based Height From Plane and Parallax

Kenneth Chaney, Alex Zihao Zhu, Kostas Daniilidis

Abstract

This work presents a novel, self-supervised framework for real-time estimation of object height and depth using event cameras on mobile platforms. By leveraging the geometric principles of "Plane and Parallax" (P+P), the proposed system achieves high-frequency (75Hz) performance and demonstrates superior robustness in challenging environments, such as high-speed driving and low-light night scenarios, where traditional frame-based cameras typically fail.

Introduction

Reliable 3D environmental perception is a cornerstone of autonomous navigation. However, conventional RGB cameras are often limited by motion blur at high velocities and restricted dynamic range in nocturnal settings. Event cameras, which asynchronously report pixel-level intensity changes, offer a hardware-level solution to these issues. Despite their advantages, interpreting sparse event data requires specialized algorithms.

Existing deep learning models for depth estimation frequently struggle with generalization, often overfitting to specific scene semantics rather than learning fundamental geometry. Moreover, dense optical flow computation is computationally prohibitive for real-time applications. This paper addresses these challenges by introducing a computationally efficient, geometry-aware method for scene structure estimation.

Methodology

The core of the approach is the Plane and Parallax (P+P) geometric framework, which simplifies the depth estimation problem by isolating off-plane motion.

Overview of the proposed Event P+P architecture, illustrating the event volume processing, ground-plane warping, and self-supervised training loop.
  1. Event Representation: Raw event streams are converted into a 3D spatiotemporal volume using trilinear interpolation, preserving critical timing information.
  2. Geometric Warping: Assuming a dominant ground plane, the system applies a homography based on camera pose to "warp" the event data. This process effectively cancels out the optical flow for any points residing on the ground plane.
  3. Residual Parallax: Objects extending above the ground plane exhibit "residual flow" in the warped space. This residual motion is geometrically linked to the object's height.
  4. Network Architecture: A convolutional neural network (CNN) is trained to predict the ratio between height ($h$) and depth ($\rho$). Predicting this ratio, rather than absolute values, enhances the model's ability to generalize across diverse environments.
  5. Self-Supervised Learning: The model is trained without LiDAR ground truth. Instead, it utilizes a photometric loss, minimizing the discrepancy between successive camera images after warping them according to the predicted scene structure.

Experimental Setup

The system was evaluated using the Multi Vehicle Stereo Event Camera (MVSEC) dataset. Training was conducted on the outdoor_day2 suburban sequence. To test generalization, evaluation was performed on outdoor_day1 (office park) and multiple outdoor_night sequences. Additionally, high-speed robustness was assessed using a motorcycle sequence reaching speeds of 140 km/hr.

Results and Discussion

The "Event P+P" method demonstrates significant improvements over baseline approaches, particularly in low-light conditions.

Quantitative comparison of Average Depth and Height Error across various MVSEC sequences. Event P+P consistently outperforms standard event-based and image-based baselines in night scenarios.

Key Performance Metrics

  • Real-Time Efficiency: The pipeline operates at 75Hz on modern GPU hardware, meeting the requirements for high-speed autonomous systems.
  • Nocturnal Superiority: In the outdoor_night1 sequence, Event P+P achieved an average depth error of 2.93m, vastly outperforming the standard Event Depth baseline of 15.62m. This highlights the advantage of event cameras' high dynamic range.
  • Freespace Detection: By thresholding the predicted heights, the system effectively identifies drivable "free space," providing a practical tool for obstacle avoidance.
  • High-Speed Robustness: Qualitative analysis of motorcycle data confirmed the system's ability to accurately detect vehicles at highway speeds where traditional cameras suffer from severe motion blur.

Conclusion

By integrating classical geometric constraints with deep learning, this work provides a robust solution for real-time 3D perception. While the method relies on a dominant ground plane assumption and precise camera calibration, it offers a significant leap in performance for autonomous vehicles operating in high-speed or low-light environments. Future research will focus on relaxing the ground plane assumption for off-road navigation and improving the handling of highly dynamic objects.

Summary updated

Abstract

Event-based cameras are a novel asynchronous sensing modality that provides exciting benefits, such as the ability to track fast moving objects with no motion blur and low latency, high dynamic range, and low power consumption. Given the low latency of the cameras, as well as their ability to work in challenging lighting conditions, these cameras are a natural fit for reactive problems such as fast local structure estimation. In this work, we propose a fast method to perform structure estimation for vehicles traveling in a roughly 2D environment (e.g. in an environment with a ground plane). Our method transfers the method of plane and parallax to events, which, given the homography to a ground plane and the pose of the camera, generates a warping of the events which removes the optical flow for events on the ground plane, while inducing flow for events above the ground plane. We then estimate dense flow in this warped space using a self-supervised neural network, which provides the height of all points in the scene. We evaluate our method on the Multi Vehicle Stereo Event Camera dataset, and show its ability to rapidly estimate the scene structure both at high speeds and in low lighting conditions.
BibTeX
@inproceedings{chaney2019learning,
  author = {Kenneth Chaney and Alex Zihao Zhu and Kostas Daniilidis},
  title = {{Learning Event-Based Height From Plane and Parallax}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  doi = {10.1109/cvprw.2019.00206},
}
Figure