Abstract
Event cameras are prized for their microsecond-level speed, but in low light, they suffer from significant "latency"—a delay between a physical change and the recorded timestamp. This paper introduces a data-driven method to correct this latency by modeling it as a function of light intensity and using a novel "temporal fidelity" metric to sharpen reconstructed images through gradient-based optimization.
Introduction: The Latency Problem in Event Cameras
Event cameras, or neuromorphic sensors, record asynchronous brightness changes ("events") rather than full frames at fixed intervals. This high temporal resolution makes them ideal for deblurring and frame interpolation. However, real-world sensors exhibit latency caused by internal electronics. While negligible in bright light, this delay can exceed 1 millisecond in low-light conditions.
Because latency varies with light intensity, events become spatially and temporally misaligned with the scene motion. This misalignment leads to ghosting artifacts and smeared edges in reconstructed images, particularly in low-light environments (e.g., 4 Lux).
Proposed Methodology
The authors propose a self-supervised framework to "undo" this delay through three key components:
1. Latency-Intensity Modeling
Controlled experiments reveal that latency is inversely related to illuminance. The system models this relationship using a $K$-th order polynomial, predicting pixel-specific delays based on the brightness of corresponding blurry frames.
2. Event-based Temporal Fidelity (ETF)
To optimize latency without ground truth, the authors introduced the ETF metric. ETF measures the sharpness and contrast of a reconstructed image. By maximizing this "no-reference" objective, the system implicitly finds the correct physical latency parameters.
3. Differentiable Event Integration
Standard reconstruction models like the Event-based Double Integral (EDI) are not naturally differentiable with respect to time. The authors reformulated the mathematics, treating events as piece-wise constant functions. This allows for gradient-based optimization (backpropagation) to learn optimal latency parameters.
Experimental Evaluation
The method was validated using synthetic data (HighREV dataset) with injected latency and real-world data captured with a DAVIS346 camera. A custom hardware setup using an Arduino-controlled light source was used to calibrate the sensor's latency-intensity curve from 30 Lux down to 0.1 Lux.
Key Results
- Deblurring: On synthetic data, correction improved PSNR by over 5% for EDI and nearly 9% for advanced models like GEM.
- Interpolation: Frame interpolation tasks saw a 17.3% PSNR improvement when paired with the REFID model.
- Visual Quality: Correction effectively removes "light strips" and ghosting, recovering sharp textures in low-light conditions that were previously smeared.
Discussion and Future Work
A key insight is that latency is spatial; dark pixels must be shifted more than bright ones. This method serves as a powerful preprocessing step for existing deep learning models, improving their performance without retraining.
Current limitations include hardware specificity (the model is calibrated for the DAVIS346) and the requirement for initial temporal synchronization between event and frame-based cameras. Future work will explore generalizing these curves across different sensor manufacturers and potential hardware-level integration.
Summary updated