Abstract
The recent years have seen a surge of interest in methods for imaging beyond the direct line of sight. The most prominent techniques rely on time-resolved optical impulse responses, obtained by illuminating a diffuse wall with an ultrashort light pulse and observing multi-bounce indirect reflections with an ultrafast time-resolved imager. Reconstruction of geometry from such data, however, is a complex non-linear inverse problem that comes with substantial computational demands. In this paper, we employ convolutional feed-forward networks for solving the reconstruction problem efficiently while maintaining good reconstruction quality. Specifically, we devise a tailored autoencoder architecture, trained end-to-end, that maps transient images directly to a depth map representation. Training is done using an efficient transient renderer for diffuse three-bounce indirect light transport that enables the quick generation of large amounts of training data for the network. We examine the performance of our method on a variety of synthetic and experimental datasets and its dependency on the choice of training data and augmentation strategies, as well as architectural features. We demonstrate that our feed-forward network, even though it is trained solely on synthetic data, generalizes to measured data from SPAD sensors and is able to obtain results that are competitive with model-based reconstruction methods.
Introduction
Non-line-of-sight (NLoS) imaging aims to reconstruct the geometry of hidden objects by firing ultrashort laser pulses at a visible surface (like a wall) and recording the multi-bounce indirect reflections using ultrafast time-resolved sensors. Reconstructing a hidden scene from this 3D "transient image"—comprising two spatial dimensions and one time-of-flight dimension—is a highly complex, non-linear inverse problem.
While deep learning has revolutionized standard computer vision, its application to NLoS imaging has been hindered by a lack of real-world training data. Capturing tens of thousands of physical NLoS measurements is practically infeasible. Furthermore, existing physics-based reconstruction methods, such as backprojection or wave-based migration, are often computationally intensive. This work addresses these challenges by proposing a neural network trained purely in simulation that successfully generalizes to real-world NLoS hardware.
Methodology
Architecture
The authors propose an end-to-end convolutional feed-forward network that maps a 3D transient image directly to a 2D depth map of the hidden scene. Due to the dimensionality mismatch between the input (3D space-time volume) and the output (2D depth map), a custom 3D/2D convolutional autoencoder is employed.
The encoder compresses the spatial and temporal dimensions using 3D convolutions. The decoder then utilizes 2D up-convolutions, followed by an upsampler and a fully-connected regressor network to produce the final depth map. This architecture allows for efficient processing of the high-dimensional transient data into a compact geometric representation.
Synthetic Training and Noise Modeling
To overcome data starvation, training data is generated using a highly efficient transient renderer simulating three-bounce indirect light transport. A critical component of the methodology is the bridging of the sim-to-real gap. Instead of a computationally expensive, high-fidelity physics simulation of the hardware, the authors use a fast, approximate Single Photon Avalanche Diode (SPAD) sensor model.
Synthetic data is augmented on the fly with Poisson noise, randomized scaling factors to simulate varying photon counts, and a global bias to account for dark counts. This simplified statistical model forces the network to learn invariant features that are robust to the noise characteristics of real-world sensors.
Datasets and Evaluation
The model's performance was evaluated using three synthetic training datasets: FlatNet (2D shapes), ShapeNet (diverse 3D models), and Redwood (real-world RGB-D crops). Evaluation was conducted on both custom synthetic test cases and publicly available experimental SPAD datasets containing complex real-world targets like statues and bicycles.
Key Results and Insights
- Real-Time Inference: The feed-forward network achieves prediction times of approximately 20 milliseconds, enabling potential real-time NLoS applications.
- Generalization: The model trained on ShapeNet demonstrated the best generalization across various datasets. Remarkably, the network successfully reconstructed real-world objects from experimental data despite being trained only on synthetic, diffuse reflections.
- Robustness: Ablation studies showed the network is resilient to significant signal degradation, including temporal downsampling and extreme shot noise.
Conclusion and Future Work
This work demonstrates that sim-to-real transfer is a viable and efficient path for NLoS reconstruction. While currently limited to a 2.5D depth map representation which cannot handle self-occlusions, future work could extend this to full 3D volumetric outputs. Additionally, combining multiple NLoS measurements from different vantage points could further enhance reconstruction quality in complex scenes.
Summary updated