Abstract
Passive Non-Line-of-Sight (NLOS) tracking enables the localization of subjects in hidden environments using ambient light and standard sensors. This paper presents PAC-Net, a deep learning framework that achieves real-time, centimeter-level tracking accuracy. By employing a dual-cell architecture that alternates between motion propagation and position calibration, PAC-Net overcomes the challenges of low signal-to-noise ratios and temporal jitter inherent in passive methods.
Introduction
Non-Line-of-Sight (NLOS) tracking—the ability to "see around corners"—has profound implications for autonomous navigation, search-and-rescue, and surveillance. Traditional active methods utilize pulsed lasers and time-of-flight sensors to achieve high precision. However, these systems are often cost-prohibitive and impractical for widespread deployment.
Passive NLOS tracking offers a low-cost alternative by analyzing ambient light reflections, such as penumbrae cast on a relay wall. Despite its potential, passive tracking faces significant hurdles: the visual signals are extremely faint (low SNR), and independent frame processing often leads to physically inconsistent, "jittery" trajectories.
The PAC-Net Architecture
To address these challenges, we introduce the Propagation and Calibration Network (PAC-Net). The architecture is designed around the principle of "predictor-corrector" methods, ensuring both temporal continuity and spatial accuracy.
Core Components
- Difference Frames: To isolate dynamic motion from static backgrounds, the system computes the difference between consecutive frames ($I_{t+1} - I_t$). This preprocessing step highlights subtle changes in the light field caused by the moving subject.
- The PAC-Cell: The heart of the network is a dual-recurrent structure:
- Propagation-Cell (P-Cell): Utilizing difference frames, the P-Cell estimates relative motion (velocity and displacement) to update the tracker's internal state.
- Calibration-Cell (C-Cell): The C-Cell processes raw frames to anchor the estimated motion to absolute coordinates, preventing the accumulation of drift over time.
- Warm-up Stage: The model undergoes an initial "warm-up" period (typically 32–64 frames). During this phase, the network internalizes the specific geometry and reflectance properties of the environment, stabilizing the hidden state before outputting trajectory data.
Data and Evaluation
The development of PAC-Net was supported by the NLOS-Track dataset, a comprehensive benchmark for dynamic passive NLOS tracking. This dataset includes:
- Real-shot Data: 500 video sequences captured with consumer-grade hardware, with ground truth provided by overhead tracking.
- Synthetic Data: Over 1,000 high-fidelity renders using Blender, simulating a wide array of lighting conditions, room layouts, and character movements.
Results and Discussion
PAC-Net demonstrates state-of-the-art performance in both accuracy and efficiency.
- Precision: The system achieves centimeter-level tracking accuracy, significantly outperforming standard CNN and RNN baselines.
- Real-Time Efficiency: PAC-Net operates at approximately 900 FPS on consumer-grade GPUs (RTX 3060) and up to 5000 FPS on high-end hardware (A100). This extreme efficiency allows for seamless integration into real-time systems.
- Robustness: The dual-cell approach effectively filters noise and maintains smooth trajectories even in challenging lighting conditions.
Conclusion and Future Work
PAC-Net represents a significant advancement in making passive NLOS tracking a practical reality. While currently limited to 2D single-object tracking, future research will focus on expanding the framework to 3D environments and multi-subject scenarios. By leveraging motion continuity as a prior, PAC-Net transforms faint shadows into precise, actionable tracking data.
Summary updated