Abstract
Aerial object tracking is a cornerstone of drone autonomy, yet most existing systems rely exclusively on RGB data, which falters in low-light or cluttered environments. Furthermore, current datasets primarily focus on high-altitude surveillance, neglecting the "overhead space" (2–5 meters) critical for human-robot interaction. This paper presents $D^2$Cube, the first large-scale benchmark for RGBD aerial tracking in overhead environments, and introduces the Efficient Multimodal Tracker (EMT). EMT leverages early modality fusion and streamlined attention mechanisms to achieve state-of-the-art accuracy while maintaining real-time performance (25 FPS) on edge computing platforms like the NVIDIA Jetson NX.
Introduction
Drone-based tracking has traditionally operated in two extremes: high-altitude surveillance (birds-eye view) or ground-level robotics. The intermediate "overhead space"—defined as altitudes between 2 and 5 meters—is increasingly important for indoor navigation, close-range inspection, and human-centric applications. In these settings, RGB-only trackers often struggle with illumination changes, motion blur, and background clutter.
While depth information (RGBD) offers a solution by providing geometric context, its integration into aerial platforms is hindered by two factors: the lack of representative datasets and the high computational overhead of multi-stream neural networks. Existing RGBD trackers are often too heavy for the limited battery and processing power of commercial drones.
The $D^2$Cube Benchmark
To address the data gap, we introduce $D^2$Cube, a massive benchmark specifically tailored for overhead aerial tracking. The dataset comprises 1,000 video sequences totaling over 1 million frames, all densely annotated with bounding boxes.
$D^2$Cube stands out for its diversity, covering 34 object categories including generic items like backpacks, bottles, and sports equipment. To ensure the benchmark's utility across different platforms, data was collected using three distinct depth sensors: Microsoft Azure Kinect, Intel RealSense D455, and ZED 2i. This variety allows researchers to evaluate how well tracking algorithms generalize across different depth-sensing technologies.
Efficient Multimodal Tracker (EMT)
The proposed Efficient Multimodal Tracker (EMT) is designed to balance tracking precision with extreme resource constraints. Its architecture avoids the traditional, computationally expensive "two-stream" approach in favor of two key innovations:
Efficient Modality-Aware Fusion (EMAF)
Rather than processing RGB and depth through separate backbones, EMT employs "early fusion." The EMAF module calculates dynamic weights ($\alpha$ and $\beta$) for each modality on a per-frame basis. This allows the system to prioritize depth in dark scenes or RGB in well-lit, high-contrast environments. By fusing these modalities into a single representation before they enter a modified ResNet-18 backbone, EMT effectively halves the backbone's computational cost.
One-Way Attention (OWA) Module
Standard transformer-based trackers suffer from quadratic complexity during feature matching. EMT introduces the OWA module, a streamlined cross-attention mechanism that performs one-way matching from the target template to the search area. By using learnable embeddings to compress template features, OWA maintains high precision without the heavy processing requirements of full self-attention.
Experimental Results
EMT was evaluated against both general RGBD trackers and specialized aerial systems. On the $D^2$Cube test set, EMT achieved a leading F-score of 0.630, significantly outperforming HCAT (0.561) and TCTrack (0.432).
Robustness in Challenging Conditions
A primary advantage of the RGBD approach is its resilience to environmental factors that cripple RGB-only systems. In scenarios involving "Dark Scenes" (DS) and "Illumination Changes" (IC), EMT maintains target lock where color-based trackers fail.
Edge Performance
Efficiency is a core metric for aerial applications. EMT utilizes only 10.05M parameters—a fraction of the 159.61M parameters used by high-end trackers like ProTrack. This optimization results in a 25x reduction in MACs. Crucially, EMT runs at 25 FPS on the NVIDIA Jetson NX Xavier, meeting the real-time requirements for closed-loop drone flight control.
Conclusion
The introduction of $D^2$Cube and EMT represents a significant step toward practical, robust aerial autonomy in human environments. By demonstrating that depth information can be integrated efficiently, this work paves the way for drones that can navigate and interact reliably in complex, real-world settings. Future work will focus on long-term occlusion handling and the integration of EMT into fully autonomous flight stacks.
Summary updated