TL;DR
This paper introduces Neural Transient Fields (NeTF), a novel neural rendering approach for Non-Line-of-Sight (NLOS) imaging. By adapting the principles of Neural Radiance Fields (NeRF) to time-of-flight measurements, NeTF can reconstruct hidden 3D scenes from sparse, time-resolved data, achieving state-of-the-art performance even with occlusions and complex materials.
What Problem Does This Paper Address?
Non-Line-of-Sight (NLOS) imaging aims to reconstruct hidden 3D scenes that are not directly visible to a sensor. Traditional methods often rely on complex inverse imaging techniques to recover scene geometry or density, which can be computationally expensive and struggle with challenges like sparse data, occlusions, and non-Lambertian surfaces. Existing approaches often require explicit 3D geometry recovery or voxel density estimation, which can be limiting. The paper addresses the need for a more robust and flexible NLOS imaging framework that can handle these challenges effectively.
Methods
The core of the NeTF approach is a neural network (MLP) that learns to represent the "neural transient field." This is inspired by NeRF, which uses an MLP to represent a scene's radiance field. However, NeTF differs in several key ways:
- Spherical Wavefronts: Instead of modeling radiance along straight lines (rays) like NeRF, NeTF models the transient signal that propagates along spherical wavefronts. This is crucial for NLOS imaging where light diffuses from a relay wall.
- Spherical Volume Rendering: A new volumetric image formation model is derived specifically for NLOS imaging under spherical coordinates.
- Sparse and Uneven Sampling: NLOS data is inherently sparse and unevenly sampled compared to the dense, regular sampling typically used in NeRF. To address this, NeTF introduces a Monte Carlo technique, specifically an MCMC approach with importance sampling, to improve reconstruction robustness.
- Two-Stage Training and Hierarchical Sampling: To combat sampling bias and improve reconstruction quality, a two-stage training strategy is employed. The first stage provides an initial reconstruction, and the second stage uses the loss from the first stage to guide resampling, focusing on areas with higher errors. Hierarchical sampling is also used to efficiently sample the wavefronts.
- Scene Representation: NeTF represents the scene as a continuous 5D function of transients, implicitly defining density and surface reflectance (albedo) without explicit voxelization.
Datasets
The paper validates NeTF on two datasets:
- Simulated ZNLOS dataset: This dataset features synthetic objects with multi-bounce transients, offering controlled ground truth.
- Real Stanford dataset: This dataset captures transients from real scenes with varying materials (diffuse, glossy, metal), providing a more challenging evaluation.
The paper does not introduce new benchmark datasets.
Key Insights and Ideas
- NeRF Principles Applied to Time-of-Flight: The paper demonstrates that the successful neural rendering paradigm of NeRF can be adapted to the distinct physics of time-of-flight measurements in NLOS imaging.
- Spherical Wavefronts are Key: Recognizing that NLOS light propagates as spherical waves, not straight rays, is fundamental to developing an accurate forward model.
- Robustness to Sparse Data: The introduction of MCMC sampling is a critical innovation for handling the sparse and uneven nature of NLOS measurements, which is a major bottleneck for many existing methods.
- Two-Stage Training for Bias Correction: The two-stage training approach effectively addresses the sampling bias inherent in NLOS data, leading to improved recovery of occluded or less visible scene parts.
- Implicit Representation for Flexibility: Representing the scene implicitly via an MLP, rather than explicit voxels, allows for arbitrary resolution reconstructions and better handling of fine details.
Key Results
- State-of-the-Art Performance: NeTF achieves state-of-the-art performance on both synthetic and real NLOS datasets, outperforming existing methods in terms of reconstruction quality.
- Improved Geometry Recovery: NeTF successfully recovers challenging geometries, such as the ears of a bunny, which are missed by previous methods.
- Robustness to Occlusions and Materials: The method demonstrates reliable reconstructions even under semi-occlusions and on non-Lambertian materials.
- Comparable Accuracy: Quantitative comparisons (e.g., Mean Absolute Error - MAE) show NeTF achieves comparable accuracy to SOTA methods, while qualitative results highlight its superior ability to preserve fine details and handle complex scenes.
- Efficiency with Sparse Data: The method shows reasonable reconstruction quality even with significantly down-sampled transient measurements (e.g., 8x8 spots).
Limitations and Future Work
- Computational Cost: While NeTF is efficient for sparse data, the training time can still be significant, comparable to NeRF, especially with higher sampling densities.
- Noise on Smooth Surfaces: The paper notes that brute-force surface extraction from the volume can lead to noise on smooth surfaces, suggesting the need for more advanced surface reconstruction techniques.
- Future Work: The authors suggest exploring emerging neural modeling techniques for smoother reconstructions and investigating acceleration schemes. They also propose tailoring NeTF to separately handle confocal and non-confocal setups, analogous to how multi-view stereo and photometric stereo are handled.
Summary updated