EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction

Burak Ercan, Onur Eker, Aykut Erdem, Erkut Erdem

Introduction

Event cameras represent a paradigm shift in optical sensing. Unlike conventional cameras that capture full frames at fixed intervals, event cameras operate asynchronously, with each pixel independently reporting changes in brightness. This mechanism offers significant advantages, including microsecond-level latency, high dynamic range (HDR), and the virtual elimination of motion blur. However, the resulting sparse, asynchronous event streams are not natively interpretable by human observers or standard computer vision algorithms designed for frame-based data.

To bridge this gap, researchers have developed various deep learning models to reconstruct intensity images (video) from event data. Despite rapid progress, the field has lacked a standardized evaluation protocol. Existing studies often utilize disparate datasets, event grouping strategies, and metrics, making side-by-side comparisons difficult. Furthermore, performance in extreme conditions—where event cameras are most beneficial—is rarely quantified due to the lack of ground-truth frames. This paper introduces EVREAL (Event-based Video Reconstruction Evaluation and Analysis Library), a unified benchmark suite designed to provide a fair and comprehensive assessment of event-to-video models.

The EVREAL Framework

EVREAL is a PyTorch-based pipeline that standardizes the entire evaluation lifecycle for event-based reconstruction. By providing a common ground for different algorithms, it allows for a rigorous comparison of seven state-of-the-art models: E2VID, FireNet, E2VID+, FireNet+, SPADE-E2VID, SSL-E2VID, and ET-Net.

The framework decomposes the reconstruction process into three standardized stages:

  1. Event Pre-processing & Grouping: This stage handles how continuous event streams are partitioned into discrete chunks, whether by fixed time windows, a fixed number of events, or synchronization with ground-truth frame timestamps.
  2. Event Representation: To facilitate processing by convolutional neural networks (CNNs), event chunks are converted into 3D voxel grids, which have become the de facto standard input format in the field.
  3. Model Inference & Post-processing: The framework executes the models and applies necessary normalizations to ensure output consistency.

Comprehensive Evaluation Methodology

A core contribution of EVREAL is its multi-faceted evaluation strategy. While standard benchmarks rely on full-reference metrics like Mean Squared Error (MSE), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), these require high-quality ground-truth frames. In challenging scenarios such as high-speed motion or extreme low light, ground-truth frames are often blurred or noisy.

To address this, EVREAL incorporates no-reference metrics (BRISQUE, NIQE, and MANIQA). These metrics assess the perceptual quality of the reconstructed video without needing a "perfect" reference image, enabling evaluation in the very conditions where event cameras excel. Additionally, the suite evaluates models on downstream tasks, including object detection, image classification, and camera calibration, to determine how reconstruction quality impacts practical utility.

Key Insights and Performance Analysis

The benchmarking process revealed several critical insights into the current state of event-based reconstruction:

1. The Leaderboard is Context-Dependent

There is no single "best" model across all categories. While ET-Net achieved the highest scores on standard full-reference metrics (e.g., an MSE of 0.047 on the ECD dataset), its performance did not always translate to downstream tasks. For instance, ET-Net performed relatively poorly when its outputs were used for object detection.

2. Robustness in Extreme Conditions

In challenging scenarios, the leaderboard often flips. Models optimized for standard datasets frequently struggle with rapid motion or low light. FireNet+ emerged as the most robust model in these environments, securing the best no-reference scores (e.g., a BRISQUE score of 18.397 on the ECD-FAST subset).

3. Downstream Task Efficacy

The utility of reconstructed video varies significantly by application:

  • Night-time Car Detection: FireNet achieved the highest Average Precision (64.11% AP).
  • Image Classification: E2VID led the field with 75.99% accuracy on N-Caltech101.
  • Camera Calibration: FireNet+ proved most accurate, with a Mean Absolute Percentage Error (MAPE) of 1.70%.

4. Efficiency vs. Quality Trade-offs

Computational overhead varies drastically. FireNet is optimized for speed, boasting a 1.6 ms inference time and only 0.04M parameters. In contrast, the high-fidelity ET-Net requires 32.1 ms per frame and contains 22.18M parameters. This ~20x speed difference highlights the gap between models suitable for real-time robotics and those designed for maximum visual fidelity.

Robustness and Limitations

EVREAL also investigates how models handle variations in event rates and tensor sparsity. The study found that robustness is inconsistent; for example, SPADE-E2VID’s performance degrades as event tensors become denser, while E2VID’s performance improves.

Currently, EVREAL is limited by its reliance on grid-based event representations. This makes it difficult to evaluate emerging architectures that utilize raw continuous event streams or graph-based representations.

Conclusion

EVREAL provides the first unified, open-source library for the systematic evaluation of event-based video reconstruction. By moving beyond simple image quality metrics and incorporating no-reference analysis and downstream task performance, it offers a more realistic picture of model capabilities. Future work will focus on adding temporal consistency metrics to address video flickering, expanding the range of downstream tasks, and supporting color video reconstruction.

Summary updated

Abstract

Event cameras are a new type of vision sensor that incorporates asynchronous and independent pixels, offering advantages over traditional frame-based cameras such as high dynamic range and minimal motion blur. However, their output is not easily understandable by humans, making the reconstruction of intensity images from event streams a fundamental task in event-based vision. While recent deep learning-based methods have shown promise in video reconstruction from events, this problem is not completely solved yet. To facilitate comparison between different approaches, standardized evaluation protocols and diverse test datasets are essential. This paper proposes a unified evaluation methodology and introduces an open-source framework called EVREAL to comprehensively benchmark and analyze various event-based video reconstruction methods from the literature. Using EVREAL, we give a detailed analysis of the state-of-the-art methods for event-based video reconstruction, and provide valuable insights into the performance of these methods under varying settings, challenging scenarios, and downstream tasks.
BibTeX
@inproceedings{ercan2023evreal,
  author = {Burak Ercan and Onur Eker and Aykut Erdem and Erkut Erdem},
  title = {{EVREAL: Towards a Comprehensive Benchmark and Analysis Suite for Event-based Video Reconstruction}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
}
Figure