Multi-Modal Non-Line-of-Sight Passive Imaging

Andre Beckus, Alexandru Tamasan, George K. Atia

TL;DR

This paper introduces a novel passive imaging technique for non-line-of-sight (NLOS) scenarios, utilizing both intensity and spatial coherence of scattered light. By formulating a multi-modal convex optimization problem and solving it with an ADMM algorithm, the method reconstructs 2D object profiles from light reflected off diffusive surfaces, even when traditional imaging fails.

What Problem Does This Paper Address?

Traditional imaging relies on direct line-of-sight between an object and a sensor. However, many real-world scenarios, such as in defense, robotics, or emergency situations, involve objects hidden from view by obstacles. Non-line-of-sight (NLOS) imaging aims to overcome this by reconstructing information about an object from light that has bounced off surrounding surfaces. Existing NLOS methods often require active control of the light source (e.g., lasers) or specific scene configurations. This paper tackles the challenge of passive NLOS imaging, where the light source is not controlled (e.g., ambient light, LEDs) and the environment might involve partially coherent light and strong surface scattering. The core problem is that under these conditions, the reflected light's intensity alone may not contain sufficient information to form a discernible image, as it can be blurred by scattering and overwhelmed by ambient light.

Methods

The authors propose a physics-driven approach that leverages both the intensity and the spatial coherence of the scattered light.

  • Physical Model: They build upon established optical models for light propagation (Fresnel model) and surface scattering. The key insight is that while intensity can be significantly degraded by scattering, the spatial coherence of the light retains valuable information about the object's shape.
  • Multi-Modal Fusion: The core of their method is a convex optimization framework that fuses measurements from two modalities:
  • Intensity Measurements: These are standard measurements of light intensity on a wall, which can capture shadow information but are susceptible to blurring and ambient light.
  • Spatial Coherence Measurements: These capture the statistical correlation of the electric field at different points on the wall. This modality is less affected by diffuse ambient light and can preserve higher-frequency details of the object's profile.
  • Convex Optimization: A multi-criteria convex optimization problem is formulated. This problem incorporates regularization terms to exploit the natural sparsity of object profiles in certain bases (specifically, the Discrete Cosine Transform - DCT). It also includes mechanisms for handling ambient light and weighting the contribution of different measurements.
  • ADMM Algorithm: To efficiently solve the complex optimization problem, the authors employ the Alternating Direction Method of Multipliers (ADMM). This iterative algorithm is well-suited for large-scale optimization problems with multiple objectives.
  • Null Space Analysis and Sample Weighting: The paper also introduces methods for analyzing the "null space" of the measurement matrices (i.e., which parts of the object's information cannot be recovered by a given measurement) and for adaptively weighting or excluding individual measurements based on their signal-to-noise ratio, improving robustness against noisy data.

Datasets

Not discussed in the paper. The paper focuses on a theoretical framework and simulated results, using synthetic data generated based on the described physical models.

Key Insights and Ideas

  • Spatial Coherence as a Key to NLOS: The most significant insight is that spatial coherence, not just intensity, is crucial for passive NLOS imaging in challenging scattering environments. While intensity can be heavily degraded, coherence retains information about the object's shape that is less affected by diffuse ambient light.
  • Fusion for Robustness: Combining intensity and coherence measurements provides a more complete picture. Intensity helps with lower-frequency details and ambient light compensation, while coherence excels at capturing higher-frequency details and is more robust to scattering.
  • Sparsity in DCT is Key: Exploiting the sparsity of object profiles in the DCT domain is essential for regularizing the inverse problem and achieving good reconstructions, especially in the presence of noise.
  • Adaptive Measurement Weighting: The ability to weight or exclude individual measurements based on their quality (e.g., SNR) is critical for handling real-world scenarios where measurements might be noisy or unreliable.

Key Results

The paper demonstrates the effectiveness of their approach through simulated experiments:

  • Coherence-Only Reconstruction: They show that using coherence measurements alone can yield discernible 2D object profiles, even with significant wall scattering that blurs intensity-based reconstructions.
  • Improved Fusion: Fusing intensity and coherence measurements consistently leads to better reconstructions than using either modality alone, as evidenced by sharper edges and more complete object profiles.
  • Noise Reduction via Sparsity: The sparsity regularization (using DCT) effectively reduces noise in the reconstructed images, particularly in the high-frequency components.
  • Sample Weighting Benefits: The proposed sample weighting scheme significantly improves reconstruction quality when dealing with noisy intensity measurements, allowing the system to rely more on cleaner coherence data.

Limitations and Future Work

  • Planar Objects: The current framework reconstructs 2D (planar) object profiles. Extending this to 3D objects is a natural next step.
  • Known Optical Distance: The method assumes the optical distance to the wall is known. The authors mention that techniques exist to estimate this distance from phase measurements, which could be integrated as a preprocessing step.
  • Single Wavelength: The current model assumes a single wavelength. Future work could explore leveraging multiple wavelengths for improved image quality.
  • Data-Driven Approaches: The null space analysis could be particularly useful in data-driven approaches where explicit physical models might not be fully known.
  • Real-World Validation: While the paper presents simulated results based on physical models, experimental validation with real-world hardware would be a crucial next step.

Summary updated

Abstract

We consider the non-line-of-sight (NLOS) imaging of an object using the light reflected off a diffusive wall. The wall scatters incident light such that a lens is no longer useful to form an image. Instead, we exploit the 4D spatial coherence function to reconstruct a 2D projection of the obscured object. The approach is completely passive in the sense that no control over the light illuminating the object is assumed and is compatible with the partially coherent fields ubiquitous in both the indoor and outdoor environments. We formulate a multi-criteria convex optimization problem for reconstruction, which fuses the reflected field's intensity and spatial coherence information at different scales. Our formulation leverages established optics models of light propagation and scattering and exploits the sparsity common to many images in different bases. We also develop an algorithm based on the alternating direction method of multipliers to efficiently solve the convex program proposed. A means for analyzing the null space of the measurement matrices is provided as well as a means for weighting the contribution of individual measurements to the reconstruction. This paper holds promise to advance passive imaging in the challenging NLOS regimes in which the intensity does not necessarily retain distinguishable features and provides a framework for multi-modal information fusion for efficient scene reconstruction.
BibTeX
@article{beckus2019multi,
  author = {Andre Beckus and Alexandru Tamasan and George K. Atia},
  title = {{Multi-Modal Non-Line-of-Sight Passive Imaging}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=cpd1ekfNin},
}
Figure