Propagate And Calibrate: Real-time Passive Non-line-of-sight Tracking

Yihao Wang, Zhigang Wang, Bin Zhao, Dong Wang, Mulin Chen, Xuelong Li

Abstract

Passive Non-Line-of-Sight (NLOS) tracking enables the localization of subjects in hidden environments using ambient light and standard sensors. This paper presents PAC-Net, a deep learning framework that achieves real-time, centimeter-level tracking accuracy. By employing a dual-cell architecture that alternates between motion propagation and position calibration, PAC-Net overcomes the challenges of low signal-to-noise ratios and temporal jitter inherent in passive methods.

Introduction

Non-Line-of-Sight (NLOS) tracking—the ability to "see around corners"—has profound implications for autonomous navigation, search-and-rescue, and surveillance. Traditional active methods utilize pulsed lasers and time-of-flight sensors to achieve high precision. However, these systems are often cost-prohibitive and impractical for widespread deployment.

Overview of the passive NLOS tracking setup.

Passive NLOS tracking offers a low-cost alternative by analyzing ambient light reflections, such as penumbrae cast on a relay wall. Despite its potential, passive tracking faces significant hurdles: the visual signals are extremely faint (low SNR), and independent frame processing often leads to physically inconsistent, "jittery" trajectories.

The PAC-Net Architecture

To address these challenges, we introduce the Propagation and Calibration Network (PAC-Net). The architecture is designed around the principle of "predictor-corrector" methods, ensuring both temporal continuity and spatial accuracy.

The PAC-Net architecture and operational stages.

Core Components

  1. Difference Frames: To isolate dynamic motion from static backgrounds, the system computes the difference between consecutive frames ($I_{t+1} - I_t$). This preprocessing step highlights subtle changes in the light field caused by the moving subject.
  2. The PAC-Cell: The heart of the network is a dual-recurrent structure:
  3. Propagation-Cell (P-Cell): Utilizing difference frames, the P-Cell estimates relative motion (velocity and displacement) to update the tracker's internal state.
  4. Calibration-Cell (C-Cell): The C-Cell processes raw frames to anchor the estimated motion to absolute coordinates, preventing the accumulation of drift over time.
  5. Warm-up Stage: The model undergoes an initial "warm-up" period (typically 32–64 frames). During this phase, the network internalizes the specific geometry and reflectance properties of the environment, stabilizing the hidden state before outputting trajectory data.

Data and Evaluation

The development of PAC-Net was supported by the NLOS-Track dataset, a comprehensive benchmark for dynamic passive NLOS tracking. This dataset includes:

  • Real-shot Data: 500 video sequences captured with consumer-grade hardware, with ground truth provided by overhead tracking.
  • Synthetic Data: Over 1,000 high-fidelity renders using Blender, simulating a wide array of lighting conditions, room layouts, and character movements.

Results and Discussion

PAC-Net demonstrates state-of-the-art performance in both accuracy and efficiency.

  • Precision: The system achieves centimeter-level tracking accuracy, significantly outperforming standard CNN and RNN baselines.
  • Real-Time Efficiency: PAC-Net operates at approximately 900 FPS on consumer-grade GPUs (RTX 3060) and up to 5000 FPS on high-end hardware (A100). This extreme efficiency allows for seamless integration into real-time systems.
  • Robustness: The dual-cell approach effectively filters noise and maintains smooth trajectories even in challenging lighting conditions.

Conclusion and Future Work

PAC-Net represents a significant advancement in making passive NLOS tracking a practical reality. While currently limited to 2D single-object tracking, future research will focus on expanding the framework to 3D environments and multi-subject scenarios. By leveraging motion continuity as a prior, PAC-Net transforms faint shadows into precise, actionable tracking data.

Summary updated

Abstract

Non-line-of-sight (NLOS) tracking has drawn increasing attention in recent years, due to its ability to detect object motion out of sight. Most previous works on NLOS tracking rely on active illumination, e.g., laser, and suffer from high cost and elaborate experimental conditions. Besides, these techniques are still far from practical application due to oversimplified settings. In contrast, we propose a purely passive method to track a person walking in an invisible room by only observing a relay wall, which is more in line with real application scenarios, e.g., security. To excavate imperceptible changes in videos of the relay wall, we introduce difference frames as an essential carrier of temporal-local motion messages. In addition, we propose PAC-Net, which consists of alternating propagation and calibration, making it capable of leveraging both dynamic and static messages on a frame-level granularity. To evaluate the proposed method, we build and publish the first dynamic passive NLOS tracking dataset, NLOS-Track, which fills the vacuum of realistic NLOS datasets. NLOS-Track contains thousands of NLOS video clips and corresponding trajectories. Both real-shot and synthetic data are included. Our codes and dataset are available at https://againstentropy.github.io/NLOS-Track/.
BibTeX
@inproceedings{wang2023propagate,
  author = {Yihao Wang and Zhigang Wang and Bin Zhao and Dong Wang and Mulin Chen and Xuelong Li},
  title = {{Propagate And Calibrate: Real-time Passive Non-line-of-sight Tracking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure