RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking

Fangwei Zhong, Xiao Bi, Yudi Zhang, Wei Zhang, Yizhou WangarXiv 2023 · DOI · arXiv

Introduction

Active Object Tracking (AOT) is a fundamental capability for autonomous systems, such as drones and mobile robots, allowing them to follow a specific target autonomously. However, maintaining a consistent track in "unstructured" environments—such as cluttered living rooms or busy parks—remains a significant challenge. Traditional end-to-end deep learning models often fail to generalize to new environments because they struggle with visual variations (lighting, textures) and the complex geometry of obstacles that can cause occlusions or collisions.

The RSPT Framework

To address these limitations, the authors introduce RSPT (Reconstruct Surroundings and Predict Trajectories). Unlike "black-box" models, RSPT employs a modular, structure-aware approach that explicitly reasons about the environment's geometry and the target's likely movement.

The RSPT framework architecture, featuring modular components for localization, mapping, and trajectory prediction.

The framework consists of four primary modules:

  1. Target Localization: Utilizes the DiMP video tracker and depth data to pinpoint the target's 3D position relative to the robot.
  2. Structure Reconstruction: Generates a lightweight, robot-centric 2D grid map in real-time, identifying navigable space and obstacles.
  3. Structure-aware Trajectory Prediction: Employs a Mixture-Density Recurrent Network (MDN-RNN) to forecast multiple potential future paths for the target, taking environmental constraints into account.
  4. Motion Controller: A Reinforcement Learning (RL) policy that translates the map and predicted trajectories into optimal robot movements.

A key innovation is the asymmetric dueling mechanism used during training. The tracker is pitted against an adversarial target designed to escape, forcing the system to learn robust recovery strategies and anticipatory behaviors.

Experimental Evaluation

The researchers validated RSPT through extensive high-fidelity simulations and real-world hardware tests. Training was conducted in the FlexibleRoom+ environment, while testing spanned six unseen simulated environments, including UrbanCity and SnowForest.

Real-world deployment on a quadruped robot, showing both third-person and first-person views during tracking.

In real-world trials, the system was deployed on a quadruped robot. Over 79 minutes of tracking (approximately 47,000 steps), RSPT demonstrated superior performance in navigating "S-type" obstacle courses and maintaining lock during visual occlusions where baseline methods typically failed.

Key Results and Insights

RSPT showed a substantial performance leap over state-of-the-art end-to-end models:

  • Success Rate: RSPT achieved a 100% success rate in UrbanCity and UrbanRoad simulations, significantly outperforming models like AD-VAT+ (40-80%).
  • Cluttered Environments: In the complex Garden environment, RSPT's accumulated reward was nearly five times higher than the best end-to-end baseline.
  • Generalization: By abstracting raw visual data into geometric grid maps, the RL policy effectively bridged the "sim-to-real" gap, as it focuses on spatial relationships rather than specific visual textures.
  • Handling Uncertainty: The MDN-RNN allows the robot to represent multi-modal futures, preventing the "averaging" of paths that often leads simpler models to collide with walls at path junctions.

Conclusion and Future Work

RSPT represents a significant advancement in making tracking robots more reliable in the real world. By combining spatial memory (via mapping) with predictive modeling, it provides robots with both a "map" and a "crystal ball."

Future research will focus on extending the 2D grid map to 3D to handle multi-level terrain (like stairs), incorporating dynamic obstacle avoidance for moving crowds, and integrating "Machine Theory of Mind" to better understand target intent for long-term forecasting.

Summary updated

Abstract

Active Object Tracking (AOT) aims to maintain a specific relation between the tracker and object(s) by autonomously controlling the motion system of a tracker given observations. AOT has wide-ranging applications, such as in mobile robots and autonomous driving. However, building a generalizable active tracker that works robustly across different scenarios remains a challenge, especially in unstructured environments with cluttered obstacles and diverse layouts. We argue that constructing a state representation capable of modeling the geometry structure of the surroundings and the dynamics of the target is crucial for achieving this goal. To address this challenge, we present RSPT, a framework that forms a structure-aware motion representation by Reconstructing the Surroundings and Predicting the target Trajectory. Additionally, we enhance the generalization of the policy network by training in an asymmetric dueling mechanism. We evaluate RSPT on various simulated scenarios and show that it outperforms existing methods in unseen environments, particularly those with complex obstacles and layouts. We also demonstrate the successful transfer of RSPT to real-world settings. Project Website: https://sites.google.com/view/aot-rspt.
BibTeX
@misc{zhong2023rspt,
  author = {Fangwei Zhong and Xiao Bi and Yudi Zhang and Wei Zhang and Yizhou Wang},
  title = {{RSPT: Reconstruct Surroundings and Predict Trajectories for Generalizable Active Object Tracking}},
  year = {2023},
  eprint = {2304.03623},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.03623},
}
Figure