Passive Ultra-Wideband Single-Photon Imaging

Mian Wei, Sotiris Nousias, Rahul Gulve, David B. Lindell, Kiriakos N. KutulakosICCV 2023 · OpenReview

Abstract

We consider the problem of imaging a dynamic scene over an extreme range of timescales simultaneously—seconds to picoseconds—and doing so passively, without much light, and without any timing signals from the light source(s) emitting it. Because existing flux estimation techniques for single-photon cameras break down in this regime, we develop a flux probing theory that draws insights from stochastic calculus to enable reconstruction of a pixel’s time-varying flux from a stream of monotonically-increasing photon detection timestamps. We use this theory to (1) show that passive free-running SPAD cameras have an attainable frequency bandwidth that spans the entire DC-to-31 GHz range in low-flux conditions, (2) derive a novel Fourier-domain flux reconstruction algorithm that scans this range for frequencies with statistically-significant support in the timestamp data, and (3) ensure the algorithm’s noise model remains valid even for very low photon counts or non-negligible dead times. We show the potential of this asynchronous imaging regime by experimentally demonstrating several never-seen-before abilities: (1) imaging a scene illuminated simultaneously by sources operating at vastly different speeds without synchronization (bulbs, projectors, multiple pulsed lasers), (2) passive non-line-of-sight video acquisition, and (3) recording ultra-wideband video, which can be played back later at 30 Hz to show everyday motions—but can also be played a billion times slower to show the propagation of light itself.

Introduction

High-speed imaging traditionally faces a fundamental "speed-light" trade-off: capturing faster phenomena requires exponentially more light. To observe ultrafast events like light propagation, current methods rely on active imaging. In these setups, a camera is synchronized in lockstep with a pulsed laser. While effective, this approach suffers from "time wrapping"—where the camera resets every pulse, blurring slow events—and requires strict control over the light source.

Researchers from the University of Toronto have introduced a new paradigm: Passive Ultra-Wideband Single-Photon Imaging. By leveraging Single-Photon Avalanche Diode (SPAD) cameras in a free-running mode, they can capture video across 12 orders of magnitude in time—from slow human motion to the picosecond flight of light—simultaneously and without synchronization.

Experimental setup with multiple unsynchronized light sources (projector, bulb, lasers) and a SPAD camera.

Flux Probing Theory

The core innovation is Flux Probing Theory, which shifts away from traditional "histogramming" (binning photons into fixed time slots). Histogramming fails when photon arrivals are sparse or when the signal frequency exceeds the binning rate.

Stochastic Calculus Framework

Instead of discrete counts, the stream of timestamps is modeled as a realization of an inhomogeneous Poisson process. Using martingales from stochastic calculus, the researchers can separate the true light flux (the signal) from the inherent Poisson noise of random photon arrivals.

Fourier Probing and CFAR Detection

The system "probes" raw timestamp data with complex exponentials (Fourier bases) to estimate frequency components directly. To navigate the billions of potential frequencies in the DC-to-GHz range, a Constant False Alarm Rate (CFAR) detector identifies frequencies with statistically significant support. This allows for efficient reconstruction even with very few photons.

The Jitter Limit

The theoretical maximum frequency is not limited by the photon rate, but by the camera's timing jitter (internal clock precision). For the hardware used, this limit is approximately 31 GHz, enabling the capture of events lasting only hundreds of picoseconds.

Experimental Results

The researchers demonstrated several unprecedented capabilities using both single-pixel scanned SPADs and 32x32 SPAD arrays.

Multi-Scale Temporal Imaging

In a single capture, the system recorded a scene containing a spinning fan (slow), a flickering smart bulb (medium), and multiple picosecond lasers (ultrafast). Because the camera maintains a continuous internal clock, it avoids time wrapping. This allows a user to "refocus" in time post-capture: playing the video at 30 Hz to see the fan, or a billion times slower to see a laser pulse propagate.

Reconstruction of time-varying flux across 9 orders of magnitude, showing zoom levels from seconds down to picoseconds.

Passive NLOS Video

The method was used for passive non-line-of-sight (NLOS) imaging. By pointing the SPAD at a wall to capture "twice-bounced" light from an out-of-sight projector, the team reconstructed recognizable video frames using only ~3,000 photons per frame.

Conclusion

This work demonstrates that passive, asynchronous imaging is fundamentally more informative than synchronized active imaging. By preserving long-term temporal context and high-frequency details, this technology could lead to "temporal microscopes" for biological imaging or advanced passive sensing for autonomous systems in complex environments.

Summary updated

Abstract

We consider the problem of imaging a dynamic scene over an extreme range of timescales simultaneously—seconds to picoseconds—and doing so passively, without much light, and without any timing signals from the light source(s) emitting it. Because existing flux estimation techniques for single-photon cameras break down in this regime, we develop a flux probing theory that draws insights from stochastic calculus to enable reconstruction of a pixel’s time-varying flux from a stream of monotonically-increasing photon detection timestamps. We use this theory to (1) show that passive free-running SPAD cameras have an attainable frequency bandwidth that spans the entire DC-to-31 GHz range in low-flux conditions, (2) derive a novel Fourier-domain flux reconstruction algorithm that scans this range for frequencies with statistically-significant support in the timestamp data, and (3) ensure the algorithm’s noise model remains valid even for very low photon counts or non-negligible dead times. We show the potential of this asynchronous imaging regime by experimentally demonstrating several never-seen-before abilities: (1) imaging a scene illuminated simultaneously by sources operating at vastly different speeds without synchronization (bulbs, projectors, multiple pulsed lasers), (2) passive non-line-of-sight video acquisition, and (3) recording ultra-wideband video, which can be played back later at 30 Hz to show everyday motions—but can also be played a billion times slower to show the propagation of light itself.
BibTeX
@inproceedings{wei2023passive,
  author = {Mian Wei and Sotiris Nousias and Rahul Gulve and David B. Lindell and Kiriakos N. Kutulakos},
  title = {{Passive Ultra-Wideband Single-Photon Imaging}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=AhjODMOZEz},
}
Figure