TL;DR
Existing non-line-of-sight (NLOS) imaging algorithms struggle to generalize to real-world captures due to hand-tuned physical priors and sensitivity to sensor noise. This paper presents a framework that incorporates two learnable physical priors—Learnable Path Compensation (LPC) to handle multi-material radiometric fall-off and an Adaptive Phasor Field (APF) for frequency-domain denoising. Trained purely on synthetic data, the model generalizes across real-world setups and low signal-to-noise ratio (SNR) captures.
What Problem Does This Paper Address?
NLOS imaging reconstructs obscured 3D objects by firing ultrafast laser pulses at a visible diffuse surface (a "relay wall") and recording multi-bounce return photons with single-photon avalanche diodes (SPADs). Translating these time-resolved measurements (transients) into 3D geometry faces two major hurdles:
- Radiometric Intensity Fall-off (RIF): Photon attenuation scales nonlinearly with distance ($1/r^2$ for retro-reflective surfaces, $1/r^4$ for diffuse ones). Real scenes contain mixed materials, but standard algorithms apply a single static attenuation exponent across the entire volume, either over-amplifying noise or under-compensating distant objects.
- Poor Generalization at Low SNR: Short exposure times and ambient light flood SPAD transient measurements with Poisson noise and high-frequency aliasing. Prior learning-based methods trained on synthetic clean data degrade significantly when tested on noisy, real-world data.
Methods
The authors build on the virtual wave phasor-field formulation, introducing two differentiable, physics-informed modules within a Learned Feature Embeddings (LFE) architecture:
Transient Measurement
──> Feature Extraction
──> Learnable Path Compensation (LPC)
──> Adaptive Phasor Field (APF)
──> Wave Propagation (RSD)
──> Rendering Net
──> Albedo, Intensity, Depth
- Learnable Path Compensation (LPC): Instead of imposing a single scalar fall-off exponent, LPC sets up three physically grounded distance-grid priors: $\{(G_Z)^1, (G_Z)^2, (G_Z)^4\}$. A 3D CNN predicts spatial probability maps over these candidate compensation profiles, producing an adaptive compensation map tailored to mixed materials and varying object depths across the scene.
- Adaptive Phasor Field (APF): In phasor-field optics, illumination functions modulate transients and act as Gaussian bandpass filters in the temporal-frequency domain. Classical methods use an empirical, fixed bandwidth. APF maps transient feature representations through 1D/spatial convolutions to predict the Gaussian window’s standard deviation ($\sigma_{\text{pred}}$), dynamically matching the filter passband ($\Delta\Omega \propto \sigma_f$) to the measurement’s SNR.
- Inference Pipeline: The compensated and filtered features pass to a Rayleigh-Sommerfeld Diffraction (RSD) wave propagation operator, followed by a rendering module that decodes 3D albedo volumes, depth maps, and 2D intensity images.
Datasets
- Synthetic Training & Seen Test Set: 2,704 training volumes and 297 test volumes derived from the LFE benchmark ($256 \times 256 \times 512$ resolution, 33 ps bin width).
- Synthetic Unseen Test Set: 100 unseen 3D objects evaluated across simulated SNR levels (48 dB, 28 dB, and 25 dB).
- Public Real-World Data: Captured transient measurements from FK (shortened to 10-minute acquisition to evaluate low SNR) and NLOST benchmarks.
- Self-Captured Real-World Benchmark: Data gathered using an in-house confocal NLOS setup (532 nm picosecond laser, 20 MHz repetition rate, Galvo mirrors, and a SPAD with ~95 ps temporal resolution). Captures comprise three multi-material scenes: ladder (retro-reflective), resolution (diffuse paper targets), and composite (mixed materials and deep depth variation), each acquired within a short 10-minute window.
Key Insights and Ideas
- Soft selection over physical bases beats raw parameter regression: Rather than letting a deep network directly regress unbounded attenuation exponents, framing LPC as a Softmax weighting over canonical physical candidates ($r^1, r^2, r^4$) speeds up training convergence while maintaining physical plausibility.
- Filter tuning inside the virtual wave domain: Real-world sensor noise primarily manifests as high-frequency transient distortion. By making the virtual illumination phasor field’s spectral width adaptive, APF performs targeted physical bandpass filtering before spatial reconstruction.
Key Results
- Synthetic Benchmark (Seen): Achieves 24.08 dB PSNR and 0.8704 SSIM on intensity, outperforming both traditional baselines (FK: 21.69 dB, RSD: 21.74 dB) and the transformer-based state-of-the-art NLOST (23.74 dB / 0.8398). It also lowers depth Mean Absolute Distance (MAD) to 0.0307.
- Compute Efficiency: Requires 16 GB GPU memory and 0.24 s inference, compared to NLOST’s 38 GB and 0.38 s.
- Robustness under Low SNR (Unseen): At 28 dB SNR, the model reaches 20.38 dB PSNR / 0.4490 SSIM, significantly outperforming NLOST (19.96 dB / 0.2469) and CNN-based I-K (16.34 dB / 0.0722).
- Qualitative Real-World Fidelity: Successfully recovers thin structural details (e.g., individual ladder rungs, chair legs, and fine text) on 10-minute and sub-minute real-world captures where competing methods output severe noise or blurred silhouettes.
Limitations and Future Work
- Confocal Assumption: The current formulation relies on confocal scanning geometry (where laser illumination and SPAD detection points coincide on the relay wall). Adapting the framework to non-confocal setups remains open.
- Sensor Noise Modeling: The synthetic training step models Poisson noise from dark counts and ambient photons, but does not capture all hardware-level non-idealities (such as SPAD timing jitter, afterpulsing, and non-uniform laser drift).
Summary updated