Towards Real-World Aerial Vision Guidance with Categorical 6D Pose Tracker

Jingtao Sun, Yaonan Wang, Danwei Wang

Abstract

Tracking the object 6-DoF pose is crucial for various downstream robot tasks and real-world applications. In this paper, we investigate the real-world robot task of aerial vision guidance for aerial robotics manipulation, utilizing category-level 6-DoF pose tracking. Aerial conditions inevitably introduce special challenges, such as rapid viewpoint changes in pitch and roll and inter-frame differences. To support these challenges in task, we firstly introduce a robust category-level 6-DoF pose tracker (Robust6DoF). This tracker leverages shape and temporal prior knowledge to explore optimal inter-frame keypoint pairs, generated under a priori structural adaptive supervision in a coarse-to-fine manner. Notably, our Robust6DoF employs a Spatial-Temporal Augmentation module to deal with the problems of the inter-frame differences and intra-class shape variations through both temporal dynamic filtering and shape-similarity filtering. We further present a Pose-Aware Discrete Servo strategy (PAD-Servo), serving as a decoupling approach to implement the final aerial vision guidance task. It contains two servo action policies to better accommodate the structural properties of aerial robotics manipulation. Exhaustive experiments on four well-known public benchmarks demonstrate the superiority of our Robust6DoF. Real-world tests directly verify that our Robust6DoF along with PAD-Servo can be readily used in real-world aerial robotic applications.

Introduction

Determining an object's 3D position and orientation in real-time—known as 6D pose tracking—is a fundamental requirement for robotic interaction. While significant progress has been made in static or ground-based environments, aerial robotics manipulation presents unique challenges. Unmanned Aerial Vehicles (UAVs) experience rapid viewpoint changes due to aggressive tilting, rolling, and pitching, which leads to perspective drift and motion blur. Furthermore, real-world applications require tracking novel objects within a category (e.g., any bottle) rather than specific instances with known CAD models. Finally, the limited onboard computational resources of drones necessitate high-efficiency, low-latency tracking solutions.

Robust6DoF and PAD-Servo Framework

To address these challenges, we propose a dual-component framework consisting of the Robust6DoF tracker and the PAD-Servo controller.

Robust6DoF Tracker

The Robust6DoF tracker utilizes a three-stage pipeline designed for resilience:

  1. 2D-3D Dense Fusion Transformer: This module employs an attention mechanism to fuse RGB pixel data with Depth point clouds, creating robust local descriptors.
  2. Spatial-Temporal Augmentation: This core module mitigates inter-frame differences and intra-class variations. It uses temporal dynamic filtering to leverage historical frame data and shape-similarity filtering to align the current observation with a generic category shape prior.
  3. Prior-Guided Keypoints: By generating 3D keypoints in a coarse-to-fine manner, the system maintains geometric consistency across shifting viewpoints.

PAD-Servo Strategy

Traditional visual servoing often struggles with the inherent instability of aerial platforms. Our Pose-Aware Discrete Servo (PAD-Servo) strategy decouples the control problem into two loops:

  • Rotational Loop: Manages the orientation of the onboard robotic arm in 3D space.
  • Translation Loop: Controls the UAV's movement within the 2D image plane.
Experimental setup and real-world guidance task process showing pose tracking of categorical objects.

Experimental Evaluation

We evaluated Robust6DoF across four major benchmarks: NOCS-REAL275, YCB-Video, YCBInEOAT, and the large-scale Wild6D dataset.

Performance and Efficiency

On the NOCS-REAL275 dataset, Robust6DoF achieved a state-of-the-art 89.8% IoU25, outperforming existing methods such as SGPA and 6-PACK. Crucially for aerial applications, the tracker maintains a high inference speed of 24.2 FPS, enabling real-time closed-loop control.

Comparison of 6-DoF pose inference speed (FPS) vs. Average Precision on IoU25 (%), highlighting Robust6DoF's superior balance of speed and accuracy.

Real-World Validation

The system was deployed on a physical quadrotor equipped with a 4-DoF manipulator. Real-world tests confirmed that the combination of Robust6DoF and PAD-Servo allows the drone to autonomously guide itself toward target objects and actively track moving ground targets, even under the dynamic conditions typical of flight.

Conclusion

Robust6DoF demonstrates that incorporating temporal knowledge and category-level shape priors significantly enhances tracking resilience in chaotic aerial environments. By decoupling flight control from manipulation through PAD-Servo, we provide a robust path toward practical aerial robotic manipulation. Future work will explore handling total occlusions and integrating multi-modal cues for more complex scene understanding.

Summary updated

Abstract

Tracking the object 6-DoF pose is crucial for various downstream robot tasks and real-world applications. In this paper, we investigate the real-world robot task of aerial vision guidance for aerial robotics manipulation, utilizing category-level 6-DoF pose tracking. Aerial conditions inevitably introduce special challenges, such as rapid viewpoint changes in pitch and roll and inter-frame differences. To support these challenges in task, we firstly introduce a robust category-level 6-DoF pose tracker (Robust6DoF). This tracker leverages shape and temporal prior knowledge to explore optimal inter-frame keypoint pairs, generated under a priori structural adaptive supervision in a coarse-to-fine manner. Notably, our Robust6DoF employs a Spatial-Temporal Augmentation module to deal with the problems of the inter-frame differences and intra-class shape variations through both temporal dynamic filtering and shape-similarity filtering. We further present a Pose-Aware Discrete Servo strategy (PAD-Servo), serving as a decoupling approach to implement the final aerial vision guidance task. It contains two servo action policies to better accommodate the structural properties of aerial robotics manipulation. Exhaustive experiments on four well-known public benchmarks demonstrate the superiority of our Robust6DoF. Real-world tests directly verify that our Robust6DoF along with PAD-Servo can be readily used in real-world aerial robotic applications.
BibTeX
@article{sun2025towards,
  author = {Jingtao Sun and Yaonan Wang and Danwei Wang},
  title = {{Towards Real-World Aerial Vision Guidance with Categorical 6D Pose Tracker}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3558237},
}
Figure