Learning an Event Sequence Embedding for Dense Event-Based Deep Stereo

Stepan Tulyakov, François Fleuret, Martin Kiefel, Peter V. Gehler, Michael HirschICCV 2019 · OpenReview

Introduction

Event-based cameras, such as the Dynamic Vision Sensor (DVS), represent a paradigm shift in computer vision. Unlike traditional cameras that capture frames at fixed intervals, event cameras report pixel-level intensity changes asynchronously. This results in a stream of events with microsecond-level temporal resolution and high dynamic range, making them ideal for high-speed motion and challenging lighting. However, the sparse and irregular nature of event data is fundamentally incompatible with standard deep learning architectures that expect synchronous, grid-based inputs. This paper introduces Deep Dense Event Stereo (DDES), a novel framework that learns to embed these asynchronous sequences into a dense representation for high-quality stereo matching.

The Continuous Fully-Connected (CFC) Layer

The core innovation of DDES is the Continuous Fully-Connected (CFC) layer. Traditional approaches to processing event data often involve discretizing events into fixed time bins or using standard fully-connected layers that treat events as a simple sequence. These methods fail to fully exploit the precise timing information provided by the sensor.

Comparison between conventional fully-connected layers and the proposed Continuous Fully-Connected (CFC) layer.

As illustrated in the figure above, a conventional FC layer assigns weights based on the discrete index of an event in a sequence. In contrast, the CFC layer employs a "kernel network"—a small multi-layer perceptron—that takes the continuous timestamp of an event as input to compute its corresponding weight. This allows the network to learn a temporal embedding where the importance of an event is a continuous function of its time, preserving the sensor's high temporal precision.

Deep Dense Event Stereo (DDES) Architecture

The DDES pipeline is designed to transform sparse event streams into dense disparity maps through a multi-stage process:

  1. Event Queueing: For every pixel, the system maintains a First-In First-Out (FIFO) queue of the $\kappa$ most recent events.
  2. Temporal Embedding via CFC: The CFC layer processes these queues to create a grid-like "event image" embedding.
  3. Stereo Matching Backbone: This embedding is fed into a state-of-the-art stereo architecture, which includes an embedding module for feature extraction, a matching module to build a cost volume, and a 3D hourglass CNN for spatial regularization.
  4. Dense Estimation: Finally, a sub-pixel estimator produces a dense disparity map.

A key insight of this work is that the spatial context provided by 3D convolutions can effectively "fill in" depth information in regions where no events occurred, enabling dense reconstruction from sparse inputs.

Experimental Results and Discussion

The researchers evaluated DDES on the Multi Vehicle Stereo Event Camera (MVSEC) dataset, focusing on the challenging "Indoor Flying" sequences. DDES achieved a Mean Depth Error (MDE) of 13.6 cm, significantly outperforming previous hand-crafted and semi-dense methods.

Qualitative comparison of DDES results against ground truth and existing semi-dense methods.

As shown in the qualitative results, DDES produces remarkably dense and smooth disparity maps that closely match the LIDAR ground truth. In comparison, previous methods like TSES or semi-dense 3D reconstructions only provide depth information at the locations of events, leaving large gaps in the scene. DDES's ability to provide a depth value for every pixel at approximately 10 frames per second makes it highly suitable for real-time robotics applications, such as autonomous drone navigation in complex environments.

Conclusion

By treating time as a continuous coordinate through the CFC layer, DDES successfully bridges the gap between asynchronous bio-inspired sensing and dense deep learning. This approach not only sets a new state-of-the-art for event-based stereo but also demonstrates the potential for learning sophisticated temporal embeddings for a wide range of event-based vision tasks.

Summary updated

Abstract

Today, a frame-based camera is the sensor of choice for machine vision applications. However, these cameras, originally developed for acquisition of static images rather than for sensing of dynamic uncontrolled visual environments, suffer from high power consumption, data rate, latency and low dynamic range. An event-based image sensor addresses these drawbacks by mimicking a biological retina. Instead of measuring the intensity of every pixel in a fixed time-interval, it reports events of significant pixel intensity changes. Every such event is represented by its position, sign of change, and timestamp, accurate to the microsecond. Asynchronous event sequences require special handling, since traditional algorithms work only with synchronous, spatially gridded data. To address this problem we introduce a new module for event sequence embedding, for use in difference applications. The module builds a representation of an event sequence by firstly aggregating information locally across time, using a novel fully-connected layer for an irregularly sampled continuous domain, and then across discrete spatial domain. Based on this module, we design a deep learning-based stereo method for event-based cameras. The proposed method is the first learning-based stereo method for an event-based camera and the only method that produces dense results. We show that large performance increases on the Multi Vehicle Stereo Event Camera Dataset (MVSEC), which became the standard set for benchmarking of event-based stereo methods.
BibTeX
@inproceedings{tulyakov2019learning,
  author = {Stepan Tulyakov and François Fleuret and Martin Kiefel and Peter V. Gehler and Michael Hirsch},
  title = {{Learning an Event Sequence Embedding for Dense Event-Based Deep Stereo}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=cxD23qs_DcX},
}
Figure