Generalizable Non-Line-of-Sight Imaging with Learnable Physical Priors

Shida Sun, Yue Li, Yueyi Zhang, Zhiwei XiongSubmission to ICLR 2025 · Rejected

TL;DR

Existing non-line-of-sight (NLOS) imaging algorithms struggle to generalize to real-world captures due to hand-tuned physical priors and sensitivity to sensor noise. This paper presents a framework that incorporates two learnable physical priors—Learnable Path Compensation (LPC) to handle multi-material radiometric fall-off and an Adaptive Phasor Field (APF) for frequency-domain denoising. Trained purely on synthetic data, the model generalizes across real-world setups and low signal-to-noise ratio (SNR) captures.

What Problem Does This Paper Address?

NLOS imaging reconstructs obscured 3D objects by firing ultrafast laser pulses at a visible diffuse surface (a "relay wall") and recording multi-bounce return photons with single-photon avalanche diodes (SPADs). Translating these time-resolved measurements (transients) into 3D geometry faces two major hurdles:

  1. Radiometric Intensity Fall-off (RIF): Photon attenuation scales nonlinearly with distance ($1/r^2$ for retro-reflective surfaces, $1/r^4$ for diffuse ones). Real scenes contain mixed materials, but standard algorithms apply a single static attenuation exponent across the entire volume, either over-amplifying noise or under-compensating distant objects.
  2. Poor Generalization at Low SNR: Short exposure times and ambient light flood SPAD transient measurements with Poisson noise and high-frequency aliasing. Prior learning-based methods trained on synthetic clean data degrade significantly when tested on noisy, real-world data.

Methods

The authors build on the virtual wave phasor-field formulation, introducing two differentiable, physics-informed modules within a Learned Feature Embeddings (LFE) architecture:

Transient Measurement 
  ──> Feature Extraction 
  ──> Learnable Path Compensation (LPC) 
  ──> Adaptive Phasor Field (APF) 
  ──> Wave Propagation (RSD) 
  ──> Rendering Net 
  ──> Albedo, Intensity, Depth
  • Learnable Path Compensation (LPC): Instead of imposing a single scalar fall-off exponent, LPC sets up three physically grounded distance-grid priors: $\{(G_Z)^1, (G_Z)^2, (G_Z)^4\}$. A 3D CNN predicts spatial probability maps over these candidate compensation profiles, producing an adaptive compensation map tailored to mixed materials and varying object depths across the scene.
  • Adaptive Phasor Field (APF): In phasor-field optics, illumination functions modulate transients and act as Gaussian bandpass filters in the temporal-frequency domain. Classical methods use an empirical, fixed bandwidth. APF maps transient feature representations through 1D/spatial convolutions to predict the Gaussian window’s standard deviation ($\sigma_{\text{pred}}$), dynamically matching the filter passband ($\Delta\Omega \propto \sigma_f$) to the measurement’s SNR.
  • Inference Pipeline: The compensated and filtered features pass to a Rayleigh-Sommerfeld Diffraction (RSD) wave propagation operator, followed by a rendering module that decodes 3D albedo volumes, depth maps, and 2D intensity images.

Datasets

  • Synthetic Training & Seen Test Set: 2,704 training volumes and 297 test volumes derived from the LFE benchmark ($256 \times 256 \times 512$ resolution, 33 ps bin width).
  • Synthetic Unseen Test Set: 100 unseen 3D objects evaluated across simulated SNR levels (48 dB, 28 dB, and 25 dB).
  • Public Real-World Data: Captured transient measurements from FK (shortened to 10-minute acquisition to evaluate low SNR) and NLOST benchmarks.
  • Self-Captured Real-World Benchmark: Data gathered using an in-house confocal NLOS setup (532 nm picosecond laser, 20 MHz repetition rate, Galvo mirrors, and a SPAD with ~95 ps temporal resolution). Captures comprise three multi-material scenes: ladder (retro-reflective), resolution (diffuse paper targets), and composite (mixed materials and deep depth variation), each acquired within a short 10-minute window.

Key Insights and Ideas

  • Soft selection over physical bases beats raw parameter regression: Rather than letting a deep network directly regress unbounded attenuation exponents, framing LPC as a Softmax weighting over canonical physical candidates ($r^1, r^2, r^4$) speeds up training convergence while maintaining physical plausibility.
  • Filter tuning inside the virtual wave domain: Real-world sensor noise primarily manifests as high-frequency transient distortion. By making the virtual illumination phasor field’s spectral width adaptive, APF performs targeted physical bandpass filtering before spatial reconstruction.

Key Results

  • Synthetic Benchmark (Seen): Achieves 24.08 dB PSNR and 0.8704 SSIM on intensity, outperforming both traditional baselines (FK: 21.69 dB, RSD: 21.74 dB) and the transformer-based state-of-the-art NLOST (23.74 dB / 0.8398). It also lowers depth Mean Absolute Distance (MAD) to 0.0307.
  • Compute Efficiency: Requires 16 GB GPU memory and 0.24 s inference, compared to NLOST’s 38 GB and 0.38 s.
  • Robustness under Low SNR (Unseen): At 28 dB SNR, the model reaches 20.38 dB PSNR / 0.4490 SSIM, significantly outperforming NLOST (19.96 dB / 0.2469) and CNN-based I-K (16.34 dB / 0.0722).
  • Qualitative Real-World Fidelity: Successfully recovers thin structural details (e.g., individual ladder rungs, chair legs, and fine text) on 10-minute and sub-minute real-world captures where competing methods output severe noise or blurred silhouettes.

Limitations and Future Work

  • Confocal Assumption: The current formulation relies on confocal scanning geometry (where laser illumination and SPAD detection points coincide on the relay wall). Adapting the framework to non-confocal setups remains open.
  • Sensor Noise Modeling: The synthetic training step models Poisson noise from dark counts and ambient photons, but does not capture all hardware-level non-idealities (such as SPAD timing jitter, afterpulsing, and non-uniform laser drift).

Summary updated

Abstract

Non-line-of-sight (NLOS) imaging, recovering the hidden volume from indirect reflections, has attracted increasing attention due to its potential applications. Despite promising results, existing NLOS reconstruction approaches are constrained by the reliance on empirical physical priors, e.g., single fixed path compensation. Moreover, these approaches still possess limited generalization ability, particularly when dealing with scenes at a low signal-to-noise ratio (SNR). To overcome the above problems, we introduce a novel learning-based approach, comprising two key designs: Learnable Path Compensation (LPC) and Adaptive Phasor Field (APF). The LPC applies tailored path compensation coefficients to adapt to different objects in the scene, effectively reducing light wave attenuation, especially in distant regions. Meanwhile, the APF learns the precise Gaussian window of the illumination function for the phasor field, dynamically selecting the relevant spectrum band of the transient measurement. Experimental validations demonstrate that our proposed approach, only trained on synthetic data, exhibits the capability to seamlessly generalize across various real-world datasets captured by different imaging systems and characterized by low SNRs.
BibTeX
@misc{sun2024generalizable,
  author = {Shida Sun and Yue Li and Yueyi Zhang and Zhiwei Xiong},
  title = {{Generalizable Non-Line-of-Sight Imaging with Learnable Physical Priors}},
  year = {2024},
  eprint = {2409.14011},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.14011},
}

Loading related content...

Figure