Latency Correction for Event-Guided Deblurring and Frame Interpolation

Yixin Yang, Jinxiu Liang, Bohan Yu, Yan Chen, Jimmy S. Ren, Boxin Shi

Abstract

Event cameras are prized for their microsecond-level speed, but in low light, they suffer from significant "latency"—a delay between a physical change and the recorded timestamp. This paper introduces a data-driven method to correct this latency by modeling it as a function of light intensity and using a novel "temporal fidelity" metric to sharpen reconstructed images through gradient-based optimization.

Introduction: The Latency Problem in Event Cameras

Event cameras, or neuromorphic sensors, record asynchronous brightness changes ("events") rather than full frames at fixed intervals. This high temporal resolution makes them ideal for deblurring and frame interpolation. However, real-world sensors exhibit latency caused by internal electronics. While negligible in bright light, this delay can exceed 1 millisecond in low-light conditions.

Comparison of ideal vs. real events showing latency and the resulting EDI deblurring before and after correction.

Because latency varies with light intensity, events become spatially and temporally misaligned with the scene motion. This misalignment leads to ghosting artifacts and smeared edges in reconstructed images, particularly in low-light environments (e.g., 4 Lux).

Proposed Methodology

The authors propose a self-supervised framework to "undo" this delay through three key components:

1. Latency-Intensity Modeling

Controlled experiments reveal that latency is inversely related to illuminance. The system models this relationship using a $K$-th order polynomial, predicting pixel-specific delays based on the brightness of corresponding blurry frames.

2. Event-based Temporal Fidelity (ETF)

To optimize latency without ground truth, the authors introduced the ETF metric. ETF measures the sharpness and contrast of a reconstructed image. By maximizing this "no-reference" objective, the system implicitly finds the correct physical latency parameters.

3. Differentiable Event Integration

Standard reconstruction models like the Event-based Double Integral (EDI) are not naturally differentiable with respect to time. The authors reformulated the mathematics, treating events as piece-wise constant functions. This allows for gradient-based optimization (backpropagation) to learn optimal latency parameters.

The Event Temporal Fidelity (ETF) metric and the differentiable event integral formulation.

Experimental Evaluation

The method was validated using synthetic data (HighREV dataset) with injected latency and real-world data captured with a DAVIS346 camera. A custom hardware setup using an Arduino-controlled light source was used to calibrate the sensor's latency-intensity curve from 30 Lux down to 0.1 Lux.

Key Results

  • Deblurring: On synthetic data, correction improved PSNR by over 5% for EDI and nearly 9% for advanced models like GEM.
  • Interpolation: Frame interpolation tasks saw a 17.3% PSNR improvement when paired with the REFID model.
  • Visual Quality: Correction effectively removes "light strips" and ghosting, recovering sharp textures in low-light conditions that were previously smeared.

Discussion and Future Work

A key insight is that latency is spatial; dark pixels must be shifted more than bright ones. This method serves as a powerful preprocessing step for existing deep learning models, improving their performance without retraining.

Current limitations include hardware specificity (the model is calibrated for the DAVIS346) and the requirement for initial temporal synchronization between event and frame-based cameras. Future work will explore generalizing these curves across different sensor manufacturers and potential hardware-level integration.

Summary updated

Abstract

Event cameras, with their high temporal resolution, dynamic range, and low power consumption, are particu-larly good at time-sensitive applications like deblurring and frame interpolation. However, their performance is hindered by latency variability, especially under low-light conditions and with fast-moving objects. This paper addresses the challenge of latency in event cameras - the temporal discrepancy between the actual occurrence of changes in the corresponding timestamp assigned by the sensor. Focusing on event-guided deblurring and frame interpolation tasks, we propose a latency correction method based on a parameterized latency model. To enable data-driven learning, we develop an event-based temporal fidelity to describe the sharpness of latent images reconstructed from events and the corresponding blurry images, and reformulate the event-based double integral model differentiable to latency. The proposed method is validated using synthetic and real-world datasets, demonstrating the benefits of latency correction for deblurring and interpolation across different lighting conditions.
BibTeX
@inproceedings{yang2024latency,
  author = {Yixin Yang and Jinxiu Liang and Bohan Yu and Yan Chen and Jimmy S. Ren and Boxin Shi},
  title = {{Latency Correction for Event-Guided Deblurring and Frame Interpolation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02359},
}
Figure