Onsite Non-Line-of-Sight Imaging via Online Calibrations

Zhengqing Pan, Ruiqian Li, Tian Gao, Zi Wang, Ping Liu, Siyuan Shen, Tao Wu, Jingyi Yu, Shiying LiarXiv 2021 · OpenReview

TL;DR

This paper introduces an "online calibration" technique for non-line-of-sight (NLOS) imaging systems. It allows for real-time recalibration of the system without needing external calibration targets, making NLOS imaging more practical for dynamic, real-world environments. This approach directly separates the light signals from the direct line-of-sight (LOS) path and the indirect NLOS path, enabling robust adjustments to scene configurations and scanning patterns.

What Problem Does This Paper Address?

Traditional non-line-of-sight (NLOS) imaging systems, which aim to "see around corners" by detecting light that has bounced off a relay surface and then off a hidden object, typically require extensive pre-calibration. This calibration process is often complex, time-consuming, and relies on specialized equipment like mirrors or checkerboards. Crucially, any changes to the experimental setup – such as moving the occluding object, altering the scanning area, or changing the scanning pattern – necessitate a full recalibration. This inflexibility makes these systems impractical for dynamic environments where configurations change frequently, such as in autonomous driving or robotics. The paper addresses this by proposing a method for "online calibration," allowing the system to adapt and recalibrate itself in situ without external aids.

Methods

The core of the proposed method is an "online calibration technique" that directly disentangles the acquired transient signals into two components:

  • Line-of-Sight (LOS) Component (Gamma Map): This component captures information about the geometry and albedo of the relay surface and the detector's position. The authors introduce the concept of a "Gamma map," which is derived from the LOS portion of the transient signals. This map is then used to directly recalibrate the system.
  • Hidden Component (NLOS): This component contains the information about the hidden object.

The system leverages a confocal imaging model, where the illumination and detection points are co-located on the relay surface. By analyzing the LOS component (Gamma map), the system can:

  • Recalibrate: Adjust for changes in scene/obstacle configurations, scanning regions, and scanning patterns.
  • Normalize Transients: Improve the quality of the captured data.
  • Preview Scanning Regions: Allow for dynamic adjustment and previewing of scanning setups.

The technique avoids auxiliary calibration apparatus like mirrors or checkerboards. It also supports both uniform and non-uniform scanning patterns, integrating with existing NLOS reconstruction algorithms. The paper details the calibration of the galvanometer (scanning mirrors) and the relay surface itself, using the Gamma map to guide these adjustments.

Datasets

The paper discusses experiments using both simulated and measured transients.

  • Simulated Transients: Synthesized for an S-shape object and a whiteboard, with and without added noise, temporal jitter, and Poisson distribution effects to mimic real-world SPAD measurements.
  • Measured Transients: Collected from various objects including a Whiteboard, S-shape, Checkerboard, Reso.board, and Mannequin. These objects are described as diffuse materials.

The paper does not introduce new benchmark datasets but rather uses existing objects and scenarios to evaluate their proposed method against state-of-the-art techniques.

Key Insights and Ideas

  • Online Calibration is Key for Practicality: The most significant insight is that real-time, "online" calibration is essential for making NLOS imaging viable in dynamic, real-world applications. The reliance on static, pre-calibrated systems is a major bottleneck.
  • Decoupling LOS and NLOS is Powerful: The ability to cleanly separate the direct (LOS) and indirect (NLOS) light paths is the foundation of their approach. The LOS component, represented by the Gamma map, becomes a powerful tool for system self-adjustment.
  • Gamma Map as a Universal Calibration Tool: The Gamma map serves multiple purposes: it calibrates the relay surface, helps define scanning regions, and normalizes transients, demonstrating its versatility.
  • Eliminating Auxiliary Calibration Targets: The method's ability to calibrate without external targets (mirrors, checkerboards) significantly simplifies the setup and makes it more portable and adaptable.
  • Support for Arbitrary Scanning Patterns: The flexibility to handle non-uniform and arbitrary scanning patterns is a notable advantage, allowing for more efficient data acquisition tailored to specific scenes.

Key Results

The paper demonstrates the effectiveness of their online calibration technique through various experiments:

  • Reconstruction Quality: The proposed optimization-based reconstruction method, when used with their calibrated system and enhanced transients, achieves comparable or superior results to state-of-the-art (SOTA) algorithms (LCT, FK, PF) on various objects.
  • Efficiency with Fewer Transients: Their method shows that reconstruction quality with their system remains similar even when using a smaller number of transients, suggesting potential for faster acquisition times.
  • Adaptability to Scanning Patterns: The system successfully reconstructs objects using different scanning patterns (regular grid, multi-circle, arbitrary), with multi-circle patterns showing slight advantages due to denser sampling in critical areas.
  • Qualitative Improvements: Figures demonstrate that their enhancement scheme improves reconstruction quality, and the online calibration allows for accurate reconstruction even with dynamic scene changes.

Limitations and Future Work

  • Hardware Alignment: While the calibration is online, the initial hardware alignment (laser and SPAD positioning) still requires manual intervention.
  • Denoising Preliminary: The transient enhancement (denoising) is described as preliminary and could be improved by accounting for more SPAD properties (e.g., Poisson distribution, pileup) and using higher-power lasers.
  • Assumptions in Reconstruction: The reconstruction algorithm assumes the hidden object is perfectly diffuse and has no occlusions or inter-reflections.
  • Future Work: The authors suggest extending their setup to support conventional NLOS imaging, potentially requiring additional devices or textured targets for estimating illumination and detection point coordinates. They also note that scanning patterns could be further optimized to align cameras and light sources in a light field imaging system for richer information and optimal acquisition time.

Summary updated

Abstract

There has been an increasing interest in deploying non-line-of-sight (NLOS) imaging systems for recovering objects behind an obstacle. Existing solutions generally pre-calibrate the system before scanning the hidden objects. Onsite adjustments of the occluder, object and scanning pattern require re-calibration. We present an online calibration technique that directly decouples the acquired transients at onsite scanning into the LOS and hidden components. We use the former to directly (re)calibrate the system upon changes of scene/obstacle configurations, scanning regions, and scanning patterns whereas the latter for hidden object recovery via spatial, frequency or learning based techniques. Our technique avoids using auxiliary calibration apparatus such as mirrors or checkerboards and supports both laboratory validations and real-world deployments.
BibTeX
@misc{pan2021onsite,
  author = {Zhengqing Pan and Ruiqian Li and Tian Gao and Zi Wang and Ping Liu and Siyuan Shen and Tao Wu and Jingyi Yu and Shiying Li},
  title = {{Onsite Non-Line-of-Sight Imaging via Online Calibrations}},
  year = {2021},
  eprint = {2112.14555},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.14555},
}

Loading related content...

Figure