TL;DR
This paper introduces a novel differentiable renderer that significantly speeds up non-line-of-sight (NLoS) reconstruction from transient images, reducing computation time from hours to minutes. This advancement enables more practical applications of NLoS imaging by making complex scene reconstruction feasible on consumer hardware and paving the way for self-supervised learning approaches.
What Problem Does This Paper Address?
Seeing around corners, or non-line-of-sight (NLoS) imaging, is a challenging problem with potential applications in areas like autonomous driving and medical imaging. The core idea involves bouncing light off a visible surface (like a wall) to illuminate a hidden object and then capturing the light's return path using sensors that can resolve time-of-flight. While existing methods can reconstruct hidden scenes from this data, they often rely on computationally expensive forward simulations of light transport, taking hours for even moderately sized scenes. This slowness, coupled with the inherent ill-posedness of the inverse problem, makes these methods impractical for many real-world scenarios.
Methods
The paper's main contribution is a fast differentiable transient renderer specifically designed for NLoS reconstruction. This renderer builds upon existing image formation models but introduces key innovations:
- Differentiable Forward Model: The renderer is formulated as a differentiable function, allowing gradients to be efficiently backpropagated. This is crucial for optimization-based reconstruction methods.
- Efficient Gradient Computation: Instead of relying on numerical approximations, the paper derives and explicitly computes gradients. This is achieved by implementing computations as NVIDIA OptiX programs for speed.
- Spatially Varying Albedo: The renderer extends previous work by allowing for spatially varying albedo (surface color) to be optimized jointly with scene geometry.
- Background Prediction Network: To handle real-world complexities like background illumination and sensor noise, a dedicated neural network is introduced to predict and subtract a transient background signal. This network is trained alongside the scene reconstruction.
- Flexible Scene Representations: The framework supports various scene representations, including radial basis functions (blobs) and depth maps, making it adaptable to different reconstruction tasks.
Datasets
The authors evaluate their method on several datasets:
- Simulated Data:
- A synthetic bunny dataset from [5] is used for quantitative and qualitative comparisons.
- A synthetic mannequin dataset from [11] is used for runtime comparisons.
- A synthetic "Spot" model with spatially varying albedo is used to demonstrate albedo reconstruction.
- Synthetic data generated from volumetric blobs is used for self-supervised learning experiments.
- Real-World Data:
- Measurements from [35] (mannequin) are used.
- Measurements from [20] (statue) are used.
- Measurements from [27] ("Diffuse S" dataset) are used.
Key Insights and Ideas
- Differentiability is Key to Speed: By making the rendering process differentiable and efficiently computing gradients, the paper demonstrates a dramatic speedup in NLoS reconstruction, moving from hours to minutes. This is a significant practical advancement.
- Joint Optimization of Geometry and Albedo: The ability to optimize for both scene shape and surface appearance simultaneously leads to more realistic and accurate reconstructions.
- Background Subtraction is Crucial for Real Data: The introduction of a background prediction network is vital for achieving good results on real-world measurements, which often suffer from unwanted ambient light.
- Flexibility for Future Work: The PyTorch implementation and the framework's adaptability make it a valuable tool for future research in NLoS imaging and potentially other differentiable rendering applications.
Key Results
- Runtime Improvement: The proposed method achieves reconstruction times in the order of minutes on consumer GPUs, a stark contrast to the hours required by previous state-of-the-art methods like [11]. For example, a synthetic scene that took 24 hours for the baseline method is reconstructed in 10 minutes by this work.
- Accuracy: The reconstructions are quantitatively comparable to or better than existing methods, achieving high Intersection over Union (IoU) scores and low Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) on benchmark datasets. For the simulated bunny dataset, their depth map reconstruction achieved the best metrics.
- Generalization: The framework demonstrates effectiveness across various scene representations (radial basis functions, depth maps) and is shown to generalize to high-resolution inputs and object tracking tasks.
- Self-Supervised Learning: The paper shows that the differentiable renderer can be used to train a reconstruction network in a self-supervised manner, highlighting its utility as a building block.
Limitations and Future Work
- Three-Bounce Assumption: The current method is restricted to three-bounce light transport, which is a necessity for achieving high performance in NLoS problems. However, this limits its applicability to scenarios where this assumption holds.
- Calibration: The authors suggest extending the software by implementing gradients with respect to scan positions to allow for calibration, similar to [17], but using more complex targets.
- Noise Model: While the L2 loss works well, the paper notes that extensions to other loss functions that more accurately represent the noise model of transient images are possible.
- Background Network Limitations: The paper acknowledges a failure case where the background network struggles with large amounts of spatially varying background, requiring manual data cleaning (flat-field correction) in some instances.
Summary updated