Introduction
Quanta Image Sensors (QIS) represent a paradigm shift in optical sensing, offering ultra-high temporal resolution (up to 100,000 fps) and zero read noise. These "super-sensors" are ideal for challenging environments such as low-light surveillance or high-speed industrial inspection. However, their adoption has been hindered by a massive "data deluge": a 1-megapixel QIS generates approximately 100 Gigabits of raw binary data every second. Processing this volume of data typically requires offline, batch-based algorithms that introduce significant latency, rendering QIS impractical for real-time applications like autonomous robotics.
The Streaming Challenge
Existing QIS processing methods are primarily "batch-based," requiring large chunks of data to be collected before reconstruction can begin. This often results in minutes of processing time for a single second of video. To bridge the gap between raw binary streams and real-time vision, a method is needed to compress the data firehose into a manageable representation without sacrificing the rich temporal information inherent in QIS data.
Methodology: The Streaming Representation
The authors propose a two-stage computational pipeline designed for extreme efficiency:
1. The "Sketch" Representation
Instead of storing every binary bit, the system maintains a "multi-exposure stack" updated via an exponential mean recursion: $R(t) = (1 - \alpha)R(t - 1) + \alpha B(t)$. By employing eight different values for $\alpha$, the system creates a stack of virtual exposures ranging from 160 microseconds to 40 milliseconds. This update requires only 16 floating-point operations (FLOPs) per pixel, allowing it to keep pace with the sensor's native 100kHz frame rate.
2. Neural Reconstruction
This 8-channel "sketch" is fed into a compact, modified U-Net with ResNet blocks. The network integrates these multi-scale temporal inputs to produce a high-quality intensity frame. Because the input dimensions are fixed, the network's runtime remains constant regardless of the sensor's actual exposure settings.
Experimental Setup and Datasets
To train and evaluate the model, the researchers utilized a combination of synthetic and real-world data:
- High-Speed Video Interpolation: The "Need for Speed" (NFS) dataset was upsampled 16x using the RIFE interpolator to simulate high-temporal resolution.
- Synthetic 3D Scenes: Complex motion and lighting were modeled using the Kubric library.
- QIS Simulation: A realistic mathematical model of SPAD-QIS sensors was applied to generate synthetic binary data.
- Real-World Validation: Testing was conducted on captures from a SwissSPAD device.
Key Results
The streaming approach yields several transformative benefits:
- Computational Efficiency: The system achieves a $10^4-10^5\times$ reduction in compute compared to previous state-of-the-art methods like Quanta Burst Photography, enabling real-time performance (10–30 FPS) on standard desktop GPUs.
- Bandwidth Reduction: Data throughput is reduced by $100\times$, making it compatible with existing hardware interfaces.
- Low Latency: The representation offers near-zero lag (under 10 microseconds), which is critical for "embodied perception" in robotics.
- Vision Task Compatibility: Reconstructed frames were successfully used with off-the-shelf models like YOLOv5 and MaskRCNN for object detection and pose estimation.
Conclusion and Future Work
By decoupling sensing from processing through a multi-scale temporal "sketch," this work enables the first real-time use of Quanta Image Sensors for computer vision. While the current model excels in low-to-medium light, future iterations may focus on high-flux scenarios and even lower latency by training vision models to operate directly on the 8-channel streaming representation.
Summary updated