Griffin: Aerial-Ground Cooperative Detection and Tracking Dataset and Benchmark

Jiahao Wang, Xiangyu Cao, Jiaru Zhong, Yuner Zhang, Haibao Yu, Lei He, Shaobing Xu

Abstract

Despite significant advancements, autonomous driving systems continue to struggle with occluded objects and long-range detection due to the inherent limitations of single-perspective sensing. Aerial-ground cooperation offers a promising solution by integrating UAVs' aerial views with ground vehicles' local observations. However, progress in this emerging field has been hindered by the absence of public datasets and standardized evaluation benchmarks. To address this gap, this paper presents a comprehensive solution for aerial-ground cooperative 3D perception through three key contributions: (1) Griffin, a large-scale multi-modal dataset featuring over 200 dynamic scenes (30k+ frames) with varied UAV altitudes (20-60m), diverse weather conditions, and occlusion-aware 3D annotations, enhanced by CARLA-AirSim co-simulation for realistic UAV dynamics; (2) A unified benchmarking framework for aerial-ground cooperative detection and tracking tasks, including protocols for evaluating communication efficiency, latency tolerance, and altitude adaptability; (3) AGILE, an instance-level intermediate fusion baseline that dynamically aligns cross-view features through query-based interaction, achieving an advantageous balance between communication overhead and perception accuracy.

Introduction

While autonomous vehicles have come a long way, single-vehicle systems still struggle with "blind spots" caused by large obstacles or limited sensor range. Cooperative perception—where vehicles share data—is a promising solution. However, traditional Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) setups are expensive and slow to deploy.

Aerial-Ground Cooperation (AGC), which uses drones as "forward scouts" for ground vehicles, offers a flexible and cost-effective alternative. Despite its potential, the field has been stuck due to a lack of representative datasets. Existing simulations often use "ideal" conditions that don't translate to the messy reality of drone flight. Furthermore, many datasets lack occlusion-aware labels, leading to noisy training signals.

Aerial-Ground Cooperation: Integrating ground-level sensing with bird's-eye views from UAVs to overcome occlusions.

The Griffin Dataset and Methodology

The authors developed a modular co-simulation framework using CARLA (for traffic and environments) and AirSim (for high-fidelity drone physics and wind modeling). The resulting Griffin Dataset consists of 255 dynamic scenes totaling over 37,000 frames and 914,000 3D annotations across four distinct maps with varying weather, lighting, and wind speeds.

Key technical contributions include:

  • Realistic Sensor Suites: The ground vehicle uses a multimodal setup (4 cameras + LiDAR), while the drone uses a vision-centric 5-camera array, reflecting real-world UAV constraints.
  • Occlusion-Aware Annotation: A visibility quantification method ensures ground truth only includes objects visible from at least one perspective, preventing models from learning from "invisible" targets.
  • Altitude Variations: Data is split into specific altitude bands (25m, 40m, 55m) and a "Random" set (20-60m) to test perspective shift handling.

Benchmarking and Insights

The paper evaluates four major fusion paradigms: Early Fusion (raw data), Intermediate Fusion (BEV-level or Instance-level), and Late Fusion (bounding boxes).

Key Insights

  • The "Altitude Gap": As drones fly higher, ground targets shrink, making them harder to detect. Current models struggle to generalize across altitudes, highlighting a need for scale-aware architectures.
  • Instance vs. Grid Fusion: Instance-level fusion (e.g., CoopTrack) proved more resilient than BEV-level grid fusion because transforming sparse 3D points is more robust to perspective distortions than aligning dense feature grids.
  • Communication Efficiency: While Early Fusion provides the best accuracy (AP 0.607), it requires 311 MB/s. Intermediate fusion methods offer a "sweet spot," providing significant gains over single-vehicle perception while staying under 1 MB/s.

Robustness Thresholds

Cooperative methods significantly outperformed single-vehicle baselines, but they are sensitive to real-world noise:

  • Latency: Performance drops below the "No Fusion" baseline once communication latency exceeds 200ms.
  • Localization: If the drone's translation error exceeds 1.5 meters, the cooperation becomes a liability, performing worse than the vehicle acting alone.

Conclusion

Griffin provides a much-needed foundation for Aerial-Ground Cooperative perception. While the gains are significant, current systems remain fragile. Future research must focus on altitude-adaptive fusion, dynamic trust mechanisms to handle noisy data, and asynchronous processing to manage real-world timing discrepancies.

The dataset and code are available at: https://github.com/wang-jh18-SVM/Griffin

Summary updated

Abstract

Despite significant advancements, autonomous driving systems continue to struggle with occluded objects and long-range detection due to the inherent limitations of single-perspective sensing. Aerial-ground cooperation offers a promising solution by integrating UAVs' aerial views with ground vehicles' local observations. However, progress in this emerging field has been hindered by the absence of public datasets and standardized evaluation benchmarks. To address this gap, this paper presents a comprehensive solution for aerial-ground cooperative 3D perception through three key contributions: (1) Griffin, a large-scale multi-modal dataset featuring over 200 dynamic scenes (30k+ frames) with varied UAV altitudes (20-60m), diverse weather conditions, and occlusion-aware 3D annotations, enhanced by CARLA-AirSim co-simulation for realistic UAV dynamics; (2) A unified benchmarking framework for aerial-ground cooperative detection and tracking tasks, including protocols for evaluating communication efficiency, latency tolerance, and altitude adaptability; (3) AGILE, an instance-level intermediate fusion baseline that dynamically aligns cross-view features through query-based interaction, achieving an advantageous balance between communication overhead and perception accuracy. Extensive experiments prove the effectiveness of aerial-ground cooperative perception and demonstrate the direction of further research. The dataset and codes are available at https://github.com/wang-jh18-SVM/Griffin.
BibTeX
@misc{wang2025griffin,
  author = {Jiahao Wang and Xiangyu Cao and Jiaru Zhong and Yuner Zhang and Haibao Yu and Lei He and Shaobing Xu},
  title = {{Griffin: Aerial-Ground Cooperative Detection and Tracking Dataset and Benchmark}},
  year = {2025},
  eprint = {2503.06983},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.06983},
}
Figure