Introduction
Event cameras represent a paradigm shift in optical sensing. Unlike conventional cameras that capture full frames at fixed intervals, event cameras operate asynchronously, with each pixel independently reporting changes in brightness. This mechanism offers significant advantages, including microsecond-level latency, high dynamic range (HDR), and the virtual elimination of motion blur. However, the resulting sparse, asynchronous event streams are not natively interpretable by human observers or standard computer vision algorithms designed for frame-based data.
To bridge this gap, researchers have developed various deep learning models to reconstruct intensity images (video) from event data. Despite rapid progress, the field has lacked a standardized evaluation protocol. Existing studies often utilize disparate datasets, event grouping strategies, and metrics, making side-by-side comparisons difficult. Furthermore, performance in extreme conditions—where event cameras are most beneficial—is rarely quantified due to the lack of ground-truth frames. This paper introduces EVREAL (Event-based Video Reconstruction Evaluation and Analysis Library), a unified benchmark suite designed to provide a fair and comprehensive assessment of event-to-video models.
The EVREAL Framework
EVREAL is a PyTorch-based pipeline that standardizes the entire evaluation lifecycle for event-based reconstruction. By providing a common ground for different algorithms, it allows for a rigorous comparison of seven state-of-the-art models: E2VID, FireNet, E2VID+, FireNet+, SPADE-E2VID, SSL-E2VID, and ET-Net.
The framework decomposes the reconstruction process into three standardized stages:
- Event Pre-processing & Grouping: This stage handles how continuous event streams are partitioned into discrete chunks, whether by fixed time windows, a fixed number of events, or synchronization with ground-truth frame timestamps.
- Event Representation: To facilitate processing by convolutional neural networks (CNNs), event chunks are converted into 3D voxel grids, which have become the de facto standard input format in the field.
- Model Inference & Post-processing: The framework executes the models and applies necessary normalizations to ensure output consistency.
Comprehensive Evaluation Methodology
A core contribution of EVREAL is its multi-faceted evaluation strategy. While standard benchmarks rely on full-reference metrics like Mean Squared Error (MSE), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), these require high-quality ground-truth frames. In challenging scenarios such as high-speed motion or extreme low light, ground-truth frames are often blurred or noisy.
To address this, EVREAL incorporates no-reference metrics (BRISQUE, NIQE, and MANIQA). These metrics assess the perceptual quality of the reconstructed video without needing a "perfect" reference image, enabling evaluation in the very conditions where event cameras excel. Additionally, the suite evaluates models on downstream tasks, including object detection, image classification, and camera calibration, to determine how reconstruction quality impacts practical utility.
Key Insights and Performance Analysis
The benchmarking process revealed several critical insights into the current state of event-based reconstruction:
1. The Leaderboard is Context-Dependent
There is no single "best" model across all categories. While ET-Net achieved the highest scores on standard full-reference metrics (e.g., an MSE of 0.047 on the ECD dataset), its performance did not always translate to downstream tasks. For instance, ET-Net performed relatively poorly when its outputs were used for object detection.
2. Robustness in Extreme Conditions
In challenging scenarios, the leaderboard often flips. Models optimized for standard datasets frequently struggle with rapid motion or low light. FireNet+ emerged as the most robust model in these environments, securing the best no-reference scores (e.g., a BRISQUE score of 18.397 on the ECD-FAST subset).
3. Downstream Task Efficacy
The utility of reconstructed video varies significantly by application:
- Night-time Car Detection: FireNet achieved the highest Average Precision (64.11% AP).
- Image Classification: E2VID led the field with 75.99% accuracy on N-Caltech101.
- Camera Calibration: FireNet+ proved most accurate, with a Mean Absolute Percentage Error (MAPE) of 1.70%.
4. Efficiency vs. Quality Trade-offs
Computational overhead varies drastically. FireNet is optimized for speed, boasting a 1.6 ms inference time and only 0.04M parameters. In contrast, the high-fidelity ET-Net requires 32.1 ms per frame and contains 22.18M parameters. This ~20x speed difference highlights the gap between models suitable for real-time robotics and those designed for maximum visual fidelity.
Robustness and Limitations
EVREAL also investigates how models handle variations in event rates and tensor sparsity. The study found that robustness is inconsistent; for example, SPADE-E2VID’s performance degrades as event tensors become denser, while E2VID’s performance improves.
Currently, EVREAL is limited by its reliance on grid-based event representations. This makes it difficult to evaluate emerging architectures that utilize raw continuous event streams or graph-based representations.
Conclusion
EVREAL provides the first unified, open-source library for the systematic evaluation of event-based video reconstruction. By moving beyond simple image quality metrics and incorporating no-reference analysis and downstream task performance, it offers a more realistic picture of model capabilities. Future work will focus on adding temporal consistency metrics to address video flickering, expanding the range of downstream tasks, and supporting color video reconstruction.
Summary updated