Resource-Efficient RGBD Aerial Tracking

Jinyu Yang, Shang Gao, Zhe Li, Feng Zheng, Ales Leonardis

Abstract

Aerial object tracking is a cornerstone of drone autonomy, yet most existing systems rely exclusively on RGB data, which falters in low-light or cluttered environments. Furthermore, current datasets primarily focus on high-altitude surveillance, neglecting the "overhead space" (2–5 meters) critical for human-robot interaction. This paper presents $D^2$Cube, the first large-scale benchmark for RGBD aerial tracking in overhead environments, and introduces the Efficient Multimodal Tracker (EMT). EMT leverages early modality fusion and streamlined attention mechanisms to achieve state-of-the-art accuracy while maintaining real-time performance (25 FPS) on edge computing platforms like the NVIDIA Jetson NX.

Introduction

Drone-based tracking has traditionally operated in two extremes: high-altitude surveillance (birds-eye view) or ground-level robotics. The intermediate "overhead space"—defined as altitudes between 2 and 5 meters—is increasingly important for indoor navigation, close-range inspection, and human-centric applications. In these settings, RGB-only trackers often struggle with illumination changes, motion blur, and background clutter.

While depth information (RGBD) offers a solution by providing geometric context, its integration into aerial platforms is hindered by two factors: the lack of representative datasets and the high computational overhead of multi-stream neural networks. Existing RGBD trackers are often too heavy for the limited battery and processing power of commercial drones.

The $D^2$Cube Benchmark

To address the data gap, we introduce $D^2$Cube, a massive benchmark specifically tailored for overhead aerial tracking. The dataset comprises 1,000 video sequences totaling over 1 million frames, all densely annotated with bounding boxes.

The $D^2$Cube dataset features 1,000 sequences across 34 categories, captured using various depth sensors to ensure cross-hardware generalization.

$D^2$Cube stands out for its diversity, covering 34 object categories including generic items like backpacks, bottles, and sports equipment. To ensure the benchmark's utility across different platforms, data was collected using three distinct depth sensors: Microsoft Azure Kinect, Intel RealSense D455, and ZED 2i. This variety allows researchers to evaluate how well tracking algorithms generalize across different depth-sensing technologies.

Efficient Multimodal Tracker (EMT)

The proposed Efficient Multimodal Tracker (EMT) is designed to balance tracking precision with extreme resource constraints. Its architecture avoids the traditional, computationally expensive "two-stream" approach in favor of two key innovations:

Efficient Modality-Aware Fusion (EMAF)

Rather than processing RGB and depth through separate backbones, EMT employs "early fusion." The EMAF module calculates dynamic weights ($\alpha$ and $\beta$) for each modality on a per-frame basis. This allows the system to prioritize depth in dark scenes or RGB in well-lit, high-contrast environments. By fusing these modalities into a single representation before they enter a modified ResNet-18 backbone, EMT effectively halves the backbone's computational cost.

One-Way Attention (OWA) Module

Standard transformer-based trackers suffer from quadratic complexity during feature matching. EMT introduces the OWA module, a streamlined cross-attention mechanism that performs one-way matching from the target template to the search area. By using learnable embeddings to compress template features, OWA maintains high precision without the heavy processing requirements of full self-attention.

Experimental Results

EMT was evaluated against both general RGBD trackers and specialized aerial systems. On the $D^2$Cube test set, EMT achieved a leading F-score of 0.630, significantly outperforming HCAT (0.561) and TCTrack (0.432).

Robustness in Challenging Conditions

A primary advantage of the RGBD approach is its resilience to environmental factors that cripple RGB-only systems. In scenarios involving "Dark Scenes" (DS) and "Illumination Changes" (IC), EMT maintains target lock where color-based trackers fail.

In low-light scenarios where RGB visibility is near zero, the depth modality provides the necessary geometric cues to maintain accurate tracking.

Edge Performance

Efficiency is a core metric for aerial applications. EMT utilizes only 10.05M parameters—a fraction of the 159.61M parameters used by high-end trackers like ProTrack. This optimization results in a 25x reduction in MACs. Crucially, EMT runs at 25 FPS on the NVIDIA Jetson NX Xavier, meeting the real-time requirements for closed-loop drone flight control.

Conclusion

The introduction of $D^2$Cube and EMT represents a significant step toward practical, robust aerial autonomy in human environments. By demonstrating that depth information can be integrated efficiently, this work paves the way for drones that can navigate and interact reliably in complex, real-world settings. Future work will focus on long-term occlusion handling and the integration of EMT into fully autonomous flight stacks.

Summary updated

Abstract

Aerial robots are now able to fly in complex environments, and drone-captured data gains lots of attention in object tracking. However, current research on aerial perception has mainly focused on limited categories, such as pedestrian or vehicle, and most scenes are captured in urban environments from a birds-eye view. Recently, UAVs equipped with depth cameras have been also deployed for more complex applications, while RGBD aerial tracking is still unexplored. Compared with traditional RGB object tracking, adding depth information can more effectively deal with more challenging scenes such as target and background interference. To this end, in this paper, we explore RGBD aerial tracking in an overhead space, which can greatly enlarge the development of drone-based visual perception. To boost the research, we first propose a large-scale benchmark for RGBD aerial tracking, containing 1,000 drone-captured RGBD videos with dense annotations. Then, as drone-based applications require for real-time processing with limited computational resources, we also propose an efficient RGBD tracker named EMT. Our tracker runs at over 100 fps on GPU, and 25 fps on the edge platform of NVidia Jetson NX Xavier, benefiting from its efficient multimodal fusion and feature matching. Extensive experiments show that our EMT achieves promising tracking performance. All resources are available at https://github.com/yjybuaa/RGBDAerialTracking.
BibTeX
@inproceedings{yang2023resource,
  author = {Jinyu Yang and Shang Gao and Zhe Li and Feng Zheng and Ales Leonardis},
  title = {{Resource-Efficient RGBD Aerial Tracking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01285},
}
Figure