Introduction
Non-Line-of-Sight (NLOS) imaging—the ability to detect and identify objects hidden from direct view—represents a significant frontier for technologies such as autonomous navigation and search-and-rescue operations. Traditional NLOS methods often rely on sophisticated, high-cost hardware like femtosecond lasers and ultra-fast detectors to measure photon time-of-flight. While conventional camera-based approaches offer a more accessible alternative, they frequently struggle in real-world scenarios where visible surfaces (e.g., walls used as "mirrors") are non-planar, textured, or occluded. This paper introduces a data-driven framework that leverages standard projectors and cameras, combined with a physics-based adaptive lighting strategy, to achieve precise 3D localization and object identification in complex environments.
Methodology
The proposed system integrates deep learning with a physics-informed lighting optimization to enhance signal detection from hidden objects.
System Pipeline
The framework consists of two primary components: a Convolutional Neural Network (CNN) for processing captured images and an adaptive lighting algorithm that optimizes the illumination of the visible scene.
Adaptive Lighting and Radiosity
The core innovation lies in the Adaptive Lighting Algorithm. Rather than uniform illumination, the system uses a radiosity-based optimization. By analyzing a 3D scan of the visible environment, the algorithm identifies optimal "patches" on the wall. Illuminating these specific areas maximizes the photon flux reflected from the hidden object back to the camera. A "Distributed Light Algorithm" further manages a limited power budget, strategically splitting light across multiple patches to maximize the signal-to-noise ratio.
CNN Architectures
The researchers developed two specialized CNN architectures based on a common backbone. One network is designed for 3D localization, outputting $(x, y, z)$ coordinates, while the other performs object identification.
Experimental Setup and Datasets
To train and validate the models, a hybrid synthetic-to-real pipeline was employed:
- Synthetic Data: 100,000 images ($64 \times 64$ resolution) were generated using the Mitsuba renderer and an "Instant Radiosity" algorithm, simulating complex geometries and textures.
- Real-World Data: A custom hardware prototype—comprising an InFocus projector and a Logitech C615 HD webcam—was used to capture 10,000 images within a 35.6 cm cube "room" containing 3D-printed objects (sphere, bunny, cylinder, human silhouette).
Results and Discussion
The integration of physics-based lighting significantly improved performance over standard illumination methods:
- Identification Accuracy: The system achieved an average accuracy of 87.1% in identifying hidden objects in real-world tests.
- Localization Precision: The mean-squared error (MSE) for 3D localization on real data was just 1.97 cm.
- Adaptive Advantage: Simulations showed that adaptive lighting improved localization accuracy by approximately 1.51 cm compared to non-optimized lighting. In real-world trials, optimal patch selection improved identification accuracy by nearly 10% over the next-best alternative.
Conclusion
This work demonstrates that combining deep learning with physics-informed adaptive lighting can overcome the limitations of traditional camera-based NLOS imaging. By concentrating light on high-yield patches, the system effectively handles complex, non-planar surfaces. Future research will focus on generalizing the model to unseen objects, handling dynamic environments, and incorporating specular reflections into the light transport model for broader real-world applicability.
Summary updated