Abstract
Active Object Tracking (AOT) in cluttered, unstructured environments remains a significant challenge for autonomous robots. Traditional methods often fail during occlusions, while end-to-end deep reinforcement learning (RL) approaches struggle with generalization to novel visual textures. We present RSPT (Reconstruct Surroundings and Predict Trajectory), a framework that enhances AOT by explicitly reconstructing local 3D geometry and predicting target trajectories as probability distributions. By decoupling visual perception from spatial reasoning, RSPT achieves superior generalization and robustness. Evaluations in high-fidelity simulations and real-world quadruped robot deployments demonstrate that RSPT significantly outperforms state-of-the-art baselines, particularly in complex, obstacle-rich scenarios.
Introduction
Active Object Tracking (AOT) requires a robot to autonomously control its movement to maintain a target within its camera's field of view. While straightforward in open spaces, AOT becomes complex in "unstructured" environments—such as cluttered living rooms or dense forests—where obstacles frequently cause occlusions.
Current solutions generally follow two paradigms: traditional two-stage pipelines and end-to-end deep RL. Traditional methods often rely on simple controllers that fail when a target is momentarily hidden. Conversely, end-to-end RL models, which map pixels directly to actions, tend to overfit to specific visual features, limiting their ability to generalize to new environments. The fundamental missing component is spatial awareness: the ability to understand environmental geometry and anticipate target movement behind obstacles.
The RSPT Framework
RSPT addresses these limitations by breaking the AOT task into four specialized, structure-aware modules:
- Target Localization: Utilizing a robust video tracker (e.g., DiMP), the system identifies the target in 2D and projects its position into 3D space using depth information.
- Structure Reconstruction: The system builds a tracker-centric local grid map (20m x 20m) from depth images. This representation identifies obstacles and navigable space, providing a geometric context that is invariant to visual textures.
- Structure-aware Trajectory Prediction: To handle uncertainty during occlusions, RSPT employs a Mixture-Density Recurrent Neural Network (MDN-RNN). This module predicts a probability distribution of the target's future path based on the local map and movement history, allowing the robot to anticipate multiple potential escape routes.
- Motion Controller: An RL-based policy determines the robot's optimal next move. To enhance robustness, the controller is trained using an asymmetric dueling mechanism, where the target is an adversarial agent attempting to evade the tracker.
Experimental Evaluation
The framework was evaluated across diverse settings:
- Simulation: Using the AD-VAT+ benchmark, RSPT was tested in six high-fidelity environments, including UrbanCity, Garden, and SnowForest.
- Real-World Deployment: The system was implemented on a quadruped robot to validate its performance against real-world sensor noise and physical constraints.
Key Results and Insights
The transition from "pixels-to-actions" to a geometry-based approach yielded significant performance gains:
- Superior Generalization: RSPT maintained a 100% success rate in four out of six unseen testing environments. In the highly cluttered "Garden" environment, RSPT achieved a 72% success rate, vastly outperforming the best end-to-end baseline, which managed only 2%.
- Robustness to Occlusion: In real-world tests involving visual occlusions, RSPT maintained a 95% success rate. Its ability to predict trajectory distributions allows it to move to strategic vantage points even when the target is temporarily out of sight.
- Geometric Invariance: By relying on a grid map rather than raw RGB data, the robot learns the logic of navigation and pursuit rather than memorizing specific environmental appearances.
Conclusion and Future Work
RSPT demonstrates that explicit geometric reconstruction and probabilistic trajectory prediction are vital for robust Active Object Tracking. While highly effective, the current implementation simplifies the 3D environment into a 2D grid, which may limit performance in environments with significant vertical complexity. Future research will explore more granular 3D representations and "Machine Theory of Mind" to better interpret and predict complex target behaviors.
Summary updated