Non-line-of-Sight Imaging via Neural Transient Fields

Siyuan Shen, Zi Wang, Ping Liu, Zhengqing Pan, Ruiqian Li, Tian Gao, Shiying Li, Jingyi Yu

TL;DR

This paper introduces Neural Transient Fields (NeTF), a novel neural rendering approach for Non-Line-of-Sight (NLOS) imaging. By adapting the principles of Neural Radiance Fields (NeRF) to time-of-flight measurements, NeTF can reconstruct hidden 3D scenes from sparse, time-resolved data, achieving state-of-the-art performance even with occlusions and complex materials.

What Problem Does This Paper Address?

Non-Line-of-Sight (NLOS) imaging aims to reconstruct hidden 3D scenes that are not directly visible to a sensor. Traditional methods often rely on complex inverse imaging techniques to recover scene geometry or density, which can be computationally expensive and struggle with challenges like sparse data, occlusions, and non-Lambertian surfaces. Existing approaches often require explicit 3D geometry recovery or voxel density estimation, which can be limiting. The paper addresses the need for a more robust and flexible NLOS imaging framework that can handle these challenges effectively.

Methods

The core of the NeTF approach is a neural network (MLP) that learns to represent the "neural transient field." This is inspired by NeRF, which uses an MLP to represent a scene's radiance field. However, NeTF differs in several key ways:

  • Spherical Wavefronts: Instead of modeling radiance along straight lines (rays) like NeRF, NeTF models the transient signal that propagates along spherical wavefronts. This is crucial for NLOS imaging where light diffuses from a relay wall.
  • Spherical Volume Rendering: A new volumetric image formation model is derived specifically for NLOS imaging under spherical coordinates.
  • Sparse and Uneven Sampling: NLOS data is inherently sparse and unevenly sampled compared to the dense, regular sampling typically used in NeRF. To address this, NeTF introduces a Monte Carlo technique, specifically an MCMC approach with importance sampling, to improve reconstruction robustness.
  • Two-Stage Training and Hierarchical Sampling: To combat sampling bias and improve reconstruction quality, a two-stage training strategy is employed. The first stage provides an initial reconstruction, and the second stage uses the loss from the first stage to guide resampling, focusing on areas with higher errors. Hierarchical sampling is also used to efficiently sample the wavefronts.
  • Scene Representation: NeTF represents the scene as a continuous 5D function of transients, implicitly defining density and surface reflectance (albedo) without explicit voxelization.

Datasets

The paper validates NeTF on two datasets:

  • Simulated ZNLOS dataset: This dataset features synthetic objects with multi-bounce transients, offering controlled ground truth.
  • Real Stanford dataset: This dataset captures transients from real scenes with varying materials (diffuse, glossy, metal), providing a more challenging evaluation.

The paper does not introduce new benchmark datasets.

Key Insights and Ideas

  • NeRF Principles Applied to Time-of-Flight: The paper demonstrates that the successful neural rendering paradigm of NeRF can be adapted to the distinct physics of time-of-flight measurements in NLOS imaging.
  • Spherical Wavefronts are Key: Recognizing that NLOS light propagates as spherical waves, not straight rays, is fundamental to developing an accurate forward model.
  • Robustness to Sparse Data: The introduction of MCMC sampling is a critical innovation for handling the sparse and uneven nature of NLOS measurements, which is a major bottleneck for many existing methods.
  • Two-Stage Training for Bias Correction: The two-stage training approach effectively addresses the sampling bias inherent in NLOS data, leading to improved recovery of occluded or less visible scene parts.
  • Implicit Representation for Flexibility: Representing the scene implicitly via an MLP, rather than explicit voxels, allows for arbitrary resolution reconstructions and better handling of fine details.

Key Results

  • State-of-the-Art Performance: NeTF achieves state-of-the-art performance on both synthetic and real NLOS datasets, outperforming existing methods in terms of reconstruction quality.
  • Improved Geometry Recovery: NeTF successfully recovers challenging geometries, such as the ears of a bunny, which are missed by previous methods.
  • Robustness to Occlusions and Materials: The method demonstrates reliable reconstructions even under semi-occlusions and on non-Lambertian materials.
  • Comparable Accuracy: Quantitative comparisons (e.g., Mean Absolute Error - MAE) show NeTF achieves comparable accuracy to SOTA methods, while qualitative results highlight its superior ability to preserve fine details and handle complex scenes.
  • Efficiency with Sparse Data: The method shows reasonable reconstruction quality even with significantly down-sampled transient measurements (e.g., 8x8 spots).

Limitations and Future Work

  • Computational Cost: While NeTF is efficient for sparse data, the training time can still be significant, comparable to NeRF, especially with higher sampling densities.
  • Noise on Smooth Surfaces: The paper notes that brute-force surface extraction from the volume can lead to noise on smooth surfaces, suggesting the need for more advanced surface reconstruction techniques.
  • Future Work: The authors suggest exploring emerging neural modeling techniques for smoother reconstructions and investigating acceleration schemes. They also propose tailoring NeTF to separately handle confocal and non-confocal setups, analogous to how multi-view stereo and photometric stereo are handled.

Summary updated

Abstract

We present a neural modeling framework for non-line-of-sight (NLOS) imaging. Previous solutions have sought to explicitly recover the 3D geometry (e.g., as point clouds) or voxel density (e.g., within a pre-defined volume) of the hidden scene. In contrast, inspired by the recent Neural Radiance Field (NeRF) approach, we use a multi-layer perceptron (MLP) to represent the neural transient field or NeTF. However, NeTF measures the transient over spherical wavefronts rather than the radiance along lines. We therefore formulate a spherical volume NeTF reconstruction pipeline, applicable to both confocal and non-confocal setups. Compared with NeRF, NeTF samples a much sparser set of viewpoints (scanning spots) and the sampling is highly uneven. We thus introduce a Monte Carlo technique to improve the robustness in the reconstruction. Experiments on synthetic and real datasets demonstrate NeTF achieves state-of-the-art performance and can provide reliable reconstructions even under semi-occlusions and on non-Lambertian materials.
BibTeX
@article{shen2021non,
  author = {Siyuan Shen and Zi Wang and Ping Liu and Zhengqing Pan and Ruiqian Li and Tian Gao and Shiying Li and Jingyi Yu},
  title = {{Non-line-of-Sight Imaging via Neural Transient Fields}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=duIgah_57Z},
}
Figure