Deep Non-Line-of-Sight Reconstruction

Javier Grau Chopite, Matthias B. Hullin, Michael Wand, Julian Iseringhausen

Abstract

The recent years have seen a surge of interest in methods for imaging beyond the direct line of sight. The most prominent techniques rely on time-resolved optical impulse responses, obtained by illuminating a diffuse wall with an ultrashort light pulse and observing multi-bounce indirect reflections with an ultrafast time-resolved imager. Reconstruction of geometry from such data, however, is a complex non-linear inverse problem that comes with substantial computational demands. In this paper, we employ convolutional feed-forward networks for solving the reconstruction problem efficiently while maintaining good reconstruction quality. Specifically, we devise a tailored autoencoder architecture, trained end-to-end, that maps transient images directly to a depth map representation. Training is done using an efficient transient renderer for diffuse three-bounce indirect light transport that enables the quick generation of large amounts of training data for the network. We examine the performance of our method on a variety of synthetic and experimental datasets and its dependency on the choice of training data and augmentation strategies, as well as architectural features. We demonstrate that our feed-forward network, even though it is trained solely on synthetic data, generalizes to measured data from SPAD sensors and is able to obtain results that are competitive with model-based reconstruction methods.

Introduction

Non-line-of-sight (NLoS) imaging aims to reconstruct the geometry of hidden objects by firing ultrashort laser pulses at a visible surface (like a wall) and recording the multi-bounce indirect reflections using ultrafast time-resolved sensors. Reconstructing a hidden scene from this 3D "transient image"—comprising two spatial dimensions and one time-of-flight dimension—is a highly complex, non-linear inverse problem.

While deep learning has revolutionized standard computer vision, its application to NLoS imaging has been hindered by a lack of real-world training data. Capturing tens of thousands of physical NLoS measurements is practically infeasible. Furthermore, existing physics-based reconstruction methods, such as backprojection or wave-based migration, are often computationally intensive. This work addresses these challenges by proposing a neural network trained purely in simulation that successfully generalizes to real-world NLoS hardware.

Methodology

Architecture

The authors propose an end-to-end convolutional feed-forward network that maps a 3D transient image directly to a 2D depth map of the hidden scene. Due to the dimensionality mismatch between the input (3D space-time volume) and the output (2D depth map), a custom 3D/2D convolutional autoencoder is employed.

The proposed 3D/2D convolutional autoencoder architecture.

The encoder compresses the spatial and temporal dimensions using 3D convolutions. The decoder then utilizes 2D up-convolutions, followed by an upsampler and a fully-connected regressor network to produce the final depth map. This architecture allows for efficient processing of the high-dimensional transient data into a compact geometric representation.

Synthetic Training and Noise Modeling

To overcome data starvation, training data is generated using a highly efficient transient renderer simulating three-bounce indirect light transport. A critical component of the methodology is the bridging of the sim-to-real gap. Instead of a computationally expensive, high-fidelity physics simulation of the hardware, the authors use a fast, approximate Single Photon Avalanche Diode (SPAD) sensor model.

Comparison of real SPAD measurements with synthetic data before and after applying the approximate noise model.

Synthetic data is augmented on the fly with Poisson noise, randomized scaling factors to simulate varying photon counts, and a global bias to account for dark counts. This simplified statistical model forces the network to learn invariant features that are robust to the noise characteristics of real-world sensors.

Datasets and Evaluation

The model's performance was evaluated using three synthetic training datasets: FlatNet (2D shapes), ShapeNet (diverse 3D models), and Redwood (real-world RGB-D crops). Evaluation was conducted on both custom synthetic test cases and publicly available experimental SPAD datasets containing complex real-world targets like statues and bicycles.

Key Results and Insights

  • Real-Time Inference: The feed-forward network achieves prediction times of approximately 20 milliseconds, enabling potential real-time NLoS applications.
  • Generalization: The model trained on ShapeNet demonstrated the best generalization across various datasets. Remarkably, the network successfully reconstructed real-world objects from experimental data despite being trained only on synthetic, diffuse reflections.
  • Robustness: Ablation studies showed the network is resilient to significant signal degradation, including temporal downsampling and extreme shot noise.

Conclusion and Future Work

This work demonstrates that sim-to-real transfer is a viable and efficient path for NLoS reconstruction. While currently limited to a 2.5D depth map representation which cannot handle self-occlusions, future work could extend this to full 3D volumetric outputs. Additionally, combining multiple NLoS measurements from different vantage points could further enhance reconstruction quality in complex scenes.

Summary updated

Abstract

The recent years have seen a surge of interest in methods for imaging beyond the direct line of sight. The most prominent techniques rely on time-resolved optical impulse responses, obtained by illuminating a diffuse wall with an ultrashort light pulse and observing multi-bounce indirect reflections with an ultrafast time-resolved imager. Reconstruction of geometry from such data, however, is a complex non-linear inverse problem that comes with substantial computational demands. In this paper, we employ convolutional feed-forward networks for solving the reconstruction problem efficiently while maintaining good reconstruction quality. Specifically, we devise a tailored autoencoder architecture, trained end-to-end, that maps transient images directly to a depth map representation. Training is done using an efficient transient renderer for diffuse three-bounce indirect light transport that enables the quick generation of large amounts of training data for the network. We examine the performance of our method on a variety of synthetic and experimental datasets and its dependency on the choice of training data and augmentation strategies, as well as architectural features. We demonstrate that our feed-forward network, even though it is trained solely on synthetic data, generalizes to measured data from SPAD sensors and is able to obtain results that are competitive with model-based reconstruction methods.
BibTeX
@inproceedings{chopite2020deep,
  author = {Javier Grau Chopite and Matthias B. Hullin and Michael Wand and Julian Iseringhausen},
  title = {{Deep Non-Line-of-Sight Reconstruction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=KrY86VsZiN8},
}
Figure