RSPT: Reconstruct Surroundings and Predict Trajectory for Generalizable Active Object Tracking

Fangwei Zhong, Xiao Bi, Yudi Zhang, Wei Zhang, Yizhou Wang

Abstract

Active Object Tracking (AOT) in cluttered, unstructured environments remains a significant challenge for autonomous robots. Traditional methods often fail during occlusions, while end-to-end deep reinforcement learning (RL) approaches struggle with generalization to novel visual textures. We present RSPT (Reconstruct Surroundings and Predict Trajectory), a framework that enhances AOT by explicitly reconstructing local 3D geometry and predicting target trajectories as probability distributions. By decoupling visual perception from spatial reasoning, RSPT achieves superior generalization and robustness. Evaluations in high-fidelity simulations and real-world quadruped robot deployments demonstrate that RSPT significantly outperforms state-of-the-art baselines, particularly in complex, obstacle-rich scenarios.

Introduction

Active Object Tracking (AOT) requires a robot to autonomously control its movement to maintain a target within its camera's field of view. While straightforward in open spaces, AOT becomes complex in "unstructured" environments—such as cluttered living rooms or dense forests—where obstacles frequently cause occlusions.

Current solutions generally follow two paradigms: traditional two-stage pipelines and end-to-end deep RL. Traditional methods often rely on simple controllers that fail when a target is momentarily hidden. Conversely, end-to-end RL models, which map pixels directly to actions, tend to overfit to specific visual features, limiting their ability to generalize to new environments. The fundamental missing component is spatial awareness: the ability to understand environmental geometry and anticipate target movement behind obstacles.

Real-world deployment of the RSPT system on a quadruped robot tracking a human target amidst obstacles.

The RSPT Framework

RSPT addresses these limitations by breaking the AOT task into four specialized, structure-aware modules:

  1. Target Localization: Utilizing a robust video tracker (e.g., DiMP), the system identifies the target in 2D and projects its position into 3D space using depth information.
  2. Structure Reconstruction: The system builds a tracker-centric local grid map (20m x 20m) from depth images. This representation identifies obstacles and navigable space, providing a geometric context that is invariant to visual textures.
  3. Structure-aware Trajectory Prediction: To handle uncertainty during occlusions, RSPT employs a Mixture-Density Recurrent Neural Network (MDN-RNN). This module predicts a probability distribution of the target's future path based on the local map and movement history, allowing the robot to anticipate multiple potential escape routes.
  4. Motion Controller: An RL-based policy determines the robot's optimal next move. To enhance robustness, the controller is trained using an asymmetric dueling mechanism, where the target is an adversarial agent attempting to evade the tracker.
The RSPT framework architecture, illustrating the integration of target localization, structure reconstruction, trajectory prediction, and motion control.

Experimental Evaluation

The framework was evaluated across diverse settings:

  • Simulation: Using the AD-VAT+ benchmark, RSPT was tested in six high-fidelity environments, including UrbanCity, Garden, and SnowForest.
  • Real-World Deployment: The system was implemented on a quadruped robot to validate its performance against real-world sensor noise and physical constraints.

Key Results and Insights

The transition from "pixels-to-actions" to a geometry-based approach yielded significant performance gains:

  • Superior Generalization: RSPT maintained a 100% success rate in four out of six unseen testing environments. In the highly cluttered "Garden" environment, RSPT achieved a 72% success rate, vastly outperforming the best end-to-end baseline, which managed only 2%.
  • Robustness to Occlusion: In real-world tests involving visual occlusions, RSPT maintained a 95% success rate. Its ability to predict trajectory distributions allows it to move to strategic vantage points even when the target is temporarily out of sight.
  • Geometric Invariance: By relying on a grid map rather than raw RGB data, the robot learns the logic of navigation and pursuit rather than memorizing specific environmental appearances.

Conclusion and Future Work

RSPT demonstrates that explicit geometric reconstruction and probabilistic trajectory prediction are vital for robust Active Object Tracking. While highly effective, the current implementation simplifies the 3D environment into a 2D grid, which may limit performance in environments with significant vertical complexity. Future research will explore more granular 3D representations and "Machine Theory of Mind" to better interpret and predict complex target behaviors.

Summary updated

Abstract

Active Object Tracking (AOT) aims to maintain a specific relation between the tracker and object(s) by autonomously controlling the motion system of a tracker given observations. It is widely used in various applications such as mobile robots and autonomous driving. However, Building a generalizable active tracker that works robustly across various scenarios remains a challenge, particularly in unstructured environments with cluttered obstacles and diverse layouts. To realize this, we argue that the key is to construct a state representation that can model the geometry structure of the surroundings and the dynamics of the target. To this end, we propose a framework called RSPT to form a structure-aware motion representation by Reconstructing Surroundings and Predicting the target Trajectory. Moreover, we further enhance the generalization of the policy network by training in the asymmetric dueling mechanism. Empirical results show that RSPT outperforms existing methods in unseen environments, especially those with cluttered obstacles and diverse layouts. We also demonstrate good sim-to-real transfer when deploying RSPT in real-world scenarios.
BibTeX
@inproceedings{zhong2023rspt,
  author = {Fangwei Zhong and Xiao Bi and Yudi Zhang and Wei Zhang and Yizhou Wang},
  title = {{RSPT: Reconstruct Surroundings and Predict Trajectory for Generalizable Active Object Tracking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25482},
}
Figure