Introduction
Event-based cameras, such as the Dynamic Vision Sensor (DVS), represent a paradigm shift in computer vision. Unlike traditional cameras that capture frames at fixed intervals, event cameras report pixel-level intensity changes asynchronously. This results in a stream of events with microsecond-level temporal resolution and high dynamic range, making them ideal for high-speed motion and challenging lighting. However, the sparse and irregular nature of event data is fundamentally incompatible with standard deep learning architectures that expect synchronous, grid-based inputs. This paper introduces Deep Dense Event Stereo (DDES), a novel framework that learns to embed these asynchronous sequences into a dense representation for high-quality stereo matching.
The Continuous Fully-Connected (CFC) Layer
The core innovation of DDES is the Continuous Fully-Connected (CFC) layer. Traditional approaches to processing event data often involve discretizing events into fixed time bins or using standard fully-connected layers that treat events as a simple sequence. These methods fail to fully exploit the precise timing information provided by the sensor.
As illustrated in the figure above, a conventional FC layer assigns weights based on the discrete index of an event in a sequence. In contrast, the CFC layer employs a "kernel network"—a small multi-layer perceptron—that takes the continuous timestamp of an event as input to compute its corresponding weight. This allows the network to learn a temporal embedding where the importance of an event is a continuous function of its time, preserving the sensor's high temporal precision.
Deep Dense Event Stereo (DDES) Architecture
The DDES pipeline is designed to transform sparse event streams into dense disparity maps through a multi-stage process:
- Event Queueing: For every pixel, the system maintains a First-In First-Out (FIFO) queue of the $\kappa$ most recent events.
- Temporal Embedding via CFC: The CFC layer processes these queues to create a grid-like "event image" embedding.
- Stereo Matching Backbone: This embedding is fed into a state-of-the-art stereo architecture, which includes an embedding module for feature extraction, a matching module to build a cost volume, and a 3D hourglass CNN for spatial regularization.
- Dense Estimation: Finally, a sub-pixel estimator produces a dense disparity map.
A key insight of this work is that the spatial context provided by 3D convolutions can effectively "fill in" depth information in regions where no events occurred, enabling dense reconstruction from sparse inputs.
Experimental Results and Discussion
The researchers evaluated DDES on the Multi Vehicle Stereo Event Camera (MVSEC) dataset, focusing on the challenging "Indoor Flying" sequences. DDES achieved a Mean Depth Error (MDE) of 13.6 cm, significantly outperforming previous hand-crafted and semi-dense methods.
As shown in the qualitative results, DDES produces remarkably dense and smooth disparity maps that closely match the LIDAR ground truth. In comparison, previous methods like TSES or semi-dense 3D reconstructions only provide depth information at the locations of events, leaving large gaps in the scene. DDES's ability to provide a depth value for every pixel at approximately 10 frames per second makes it highly suitable for real-time robotics applications, such as autonomous drone navigation in complex environments.
Conclusion
By treating time as a continuous coordinate through the CFC layer, DDES successfully bridges the gap between asynchronous bio-inspired sensing and dense deep learning. This approach not only sets a new state-of-the-art for event-based stereo but also demonstrates the potential for learning sophisticated temporal embeddings for a wide range of event-based vision tasks.
Summary updated