Abstract: Researchers from the University of Zaragoza and the University of Wisconsin–Madison have developed a framework to estimate the Light Transport Matrix (LTM) for scenes hidden around corners. By leveraging "phasor fields" to create virtual projector-camera setups on a relay wall, this method enables advanced capabilities such as relighting hidden scenes and separating direct light from complex indirect reflections, bridging a critical gap between traditional line-of-sight imaging and non-line-of-sight (NLOS) techniques.
Introduction
Non-Line-of-Sight (NLOS) imaging, which utilizes ultra-fast lasers to "see" objects hidden around corners by reflecting light off a relay wall, has traditionally focused on basic 3D reconstruction. However, it has lacked the sophisticated light analysis tools available to traditional Line-of-Sight (LOS) imaging. This research introduces a computational framework for virtual Light Transport Matrices (LTMs), porting a classic tool from computer vision into the NLOS domain. This allows researchers to not just see hidden objects, but to manipulate and analyze the light interacting with them.
The NLOS Imaging Challenge
In traditional LOS imaging, the LTM is a fundamental tool that describes how every point in a scene responds to light from every possible source. With an LTM, one can perform "dual photography," relight a scene post-capture, or separate "direct" light (hitting an object once) from "indirect" light (bouncing between objects).
Applying these techniques directly to NLOS is difficult because the relay wall acts like a massive, wide-aperture lens. This results in extreme blur (defocus) and significant signal cross-talk. While a large relay wall provides high light-gathering power, the resulting shallow depth-of-field makes isolating specific light paths computationally challenging.
Computational Framework for Virtual LTMs
The authors bridge this gap by treating the diffuse relay wall as a virtual optical system. The framework relies on several key innovations:
- Phasor Fields: Building on the phasor fields framework, the system treats the propagation of transient light (time-of-flight data) as a wave-based problem. This allows the mathematical "projection" of virtual light sources and the placement of virtual cameras onto the relay wall.
- Temporal Gating (Temporal Focusing): To overcome wide-aperture blur, the researchers use time-of-flight gating. By considering only light that returns within a specific time window, they can effectively "focus" the virtual camera and projector on a specific voxel in 3D space, stripping away out-of-focus light.
- The Geometry Oracle: The system uses calculated "direct" NLOS light (the 3-bounce path: Laser → Wall → Object → Wall → Sensor) as a mask. This "oracle" identifies actual geometry, preventing noise from empty space.
Experimental Validation and Results
The researchers validated their approach using both simulated transient data (e.g., a crowded living room) and real-world data captured with a 532nm pulsed laser and a SPAD array scanning a $24 \times 24$ grid.
The framework successfully demonstrated the ability to computationally relight specific locations of a hidden scene. Most importantly, it enabled Component Separation: the ability to isolate direct reflections from first-order indirect (two bounces within the hidden scene) and higher-order indirect illumination.
As shown in the results, the virtual LTM can distinguish the "direct" three-bounce path from complex four-bounce (or more) paths, which is essential for understanding cluttered hidden environments. The NLOS-derived indirect light components showed high qualitative agreement with LOS ground-truth simulations.
Discussion and Future Work
While the framework is robust and hardware-agnostic—scaling from 1D SPAD arrays to single-point sensors—it faces computational hurdles. Currently, the LTM is computed via brute-force sampling, which is computationally expensive ($O(K_v^2)$ relative to the number of voxels). Future work will explore matrix compression techniques, such as low-rank approximations, to speed up acquisition and develop more sophisticated functions to isolate even higher-order light bounces (5th bounce and beyond).
Summary updated