Introduction
Capturing high-speed motion in low-light environments remains a significant hurdle in computer vision. Conventional CMOS sensors face a fundamental trade-off: long exposure times gather sufficient light but introduce severe motion blur, while short exposures minimize blur at the cost of high noise levels. While specialized sensors like Single-Photon Avalanche Diodes (SPADs) and Event Cameras offer potential solutions, they introduce their own challenges—primarily a massive data bandwidth requirement for SPADs and a lack of absolute intensity information for event cameras.
This research introduces a novel sensor fusion framework that combines the microsecond precision of SPADs with the low-bandwidth efficiency of event cameras. By using events to "guide" and deblur SPAD data, the authors achieve high-quality, 100 kHz image reconstruction in near-darkness while reducing data bandwidth by up to 1000x compared to existing methods.
Methodology
The proposed framework addresses the bandwidth-performance trade-off through three core components:
- Nonlinear Event Double Integration (NEDI): Unlike traditional linear deblurring, NEDI accounts for the unique non-linear response function of SPAD sensors. It uses high-temporal-resolution event data to sharpen aggregated, blurry SPAD frames.
- Asynchronous Kalman Filter (AKF): This filter treats events and deblurred SPAD frames as asynchronous updates to reconstruct a continuous video stream. It incorporates sophisticated noise models, including shot noise and sensor refractory periods.
- Adaptive SPAD Sampling: To optimize bandwidth, the system triggers SPAD frames based on the uncertainty estimate from the Kalman filter. When the event camera provides sufficient information, the SPAD remains inactive, saving power and data.
Experimental Evaluation
The approach was validated using SimSPAD, a high-fidelity synthetic dataset, and HS-ESPAD, a real-world hardware rig featuring a SwissSPAD2 sensor and a Prophesee Gen4 event camera. Testing included challenging scenarios such as rapidly rotating fans and high-contrast targets in low-light settings.
Key Results and Insights
The fusion of these sensors leverages their complementary physics: SPADs provide absolute intensity ("what"), while event cameras provide precise motion timing ("where/when"). Key achievements include:
- Enhanced SNR: A 5 dB PSNR improvement in low-light reconstruction at 100 kHz over conventional cameras.
- Extreme Efficiency: A 1000x reduction in bandwidth and power requirements compared to SPAD-only baselines like Quanta Burst Photography.
- HDR Capability: Successful reconstruction of details in both dark indoor and bright outdoor regions simultaneously.
Conclusion and Future Work
While the current system demonstrates a breakthrough in high-speed, low-light imaging, challenges remain in sensor calibration and hardware integration. Future work aims to develop a single hybrid chip to perform this fusion at the hardware level, further pushing the boundaries of imaging in the dark.
Summary updated