Deep Event Stereo Leveraged by Event-to-Image Translation

Soikat Hasan Ahmed, Hae Woong Jang, S. M. Nadim Uddin, Yong Ju JungAAAI 2021 · DOI

Introduction

Event cameras, or Dynamic Vision Sensors (DVS), represent a paradigm shift in computer vision. Unlike traditional frame-based cameras that capture images at fixed intervals, event cameras report per-pixel intensity changes asynchronously. This allows for microsecond-level latency, high dynamic range (HDR), and the virtual elimination of motion blur. These properties make them ideal for high-speed robotics and challenging lighting conditions.

However, the output of an event camera is spatially sparse. Because they only record changes, static scenes produce no data, and moving objects are represented primarily by their edges. This sparsity makes dense depth estimation—specifically stereo matching—exceptionally difficult. Traditional methods often produce "blobby" depth maps that lack structural detail on texture-less surfaces. This paper addresses this by using event-to-image translation to provide the necessary spatial guidance for accurate stereo matching.

Proposed Methodology

The authors introduce a four-stage pipeline designed to transform sparse event streams into dense, high-accuracy disparity maps.

The overall pipeline of the proposed deep event stereo framework, illustrating the flow from raw event queues to the final disparity map.

1. Event Embedding

The process begins with Event Embedding, where raw event queues from the left and right cameras are converted into dense feature representations. This stage utilizes a kernel network with continuous fully connected layers to bridge the gap between asynchronous event points and structured convolutional grids.

2. Image Reconstruction Sub-network

A significant contribution of this work is the Image Reconstruction Sub-network. This module translates the embedded event features into reconstructed intensity images. It employs a dual-path encoder-decoder architecture with a Cross-Semantic Attention mechanism. This mechanism reweights features using both spatial and channel-wise descriptors, allowing the network to recover the scene's structural "priors"—such as the continuity of a wall or the boundary of an object—that are often hidden within the sparse event data.

3. Feature Aggregation Sub-network

Once the intensity images are reconstructed, their features are used to guide the original event features. This is handled by the Feature Aggregation Sub-network using Stacked Dilated SPADE (Spatially-Adaptive Denormalization).

The Feature Aggregation Module, which uses reconstructed image features (f_image) to modulate and guide the embedded event features (f_event).

In this stage, the reconstructed image features act as a spatial template, scaling and shifting the event features to inject contextual information. The inclusion of dilated convolutions within the SPADE blocks provides a larger receptive field, enabling the model to use global scene structure to resolve local matching ambiguities in the sparse data.

4. Stereo Matching Sub-network

The final stage is the Stereo Matching Sub-network. The fused, image-guided features from both camera views are fed into a cost-volume-based matching network. Because these features now contain rich structural information derived from the reconstruction process, the resulting disparity map is significantly more accurate and dense than those produced by methods relying solely on raw events.

Experimental Results and Key Insights

The model was rigorously evaluated on the MVSEC (Multi Vehicle Stereo Event Camera) dataset, the industry standard for event-based depth estimation.

Quantitative Performance

The proposed method achieved a new state-of-the-art on the "Indoor Flying" sequences. Compared to the previous leading method, DDES (Deep Dense Event Stereo), the Mean Depth Error was reduced from 16.6 cm to 14.2 cm. Furthermore, the One-pixel accuracy (1PA)—the percentage of pixels matched within a one-pixel margin—rose from 89.8% to 92.1%.

Qualitative Improvements

Visually, the impact of the image-translation guidance is clear. While previous methods often produced blurred or noisy depth maps on flat, texture-less surfaces, the proposed method generates sharp edges and plausible structures. This confirms the core insight of the research: even a reconstructed image contains vital geometric information that raw events lack.

Conclusion

By integrating event-to-image translation as a core guidance mechanism, this research provides a robust framework for dense depth estimation with event cameras. While the computational overhead of the reconstruction sub-network remains a consideration for real-time edge applications, the modularity of the approach allows it to be adapted to various stereo matching backbones. Future work will likely focus on improving the fidelity of the reconstructed images and testing the system in high-speed outdoor automotive environments.

Summary updated

BibTeX
@inproceedings{ahmed2021deep,
  author = {Soikat Hasan Ahmed and Hae Woong Jang and S. M. Nadim Uddin and Yong Ju Jung},
  title = {{Deep Event Stereo Leveraged by Event-to-Image Translation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i2.16171},
}
Figure