SynDroneVision: A Synthetic Dataset for Image-Based Drone Detection

Tamara R. Lenhard, Andreas Weinmann, Kai Franke, Tobias KochWACV 2025

TL;DR

This paper introduces SynDroneVision, a large-scale synthetic dataset for RGB-based drone detection, addressing the critical need for diverse and cost-effective training data. By leveraging advanced game engine capabilities, SynDroneVision significantly improves the robustness and performance of drone detection models, especially when combined with real-world data.

What Problem Does This Paper Address?

The proliferation of Unmanned Aerial Vehicles (UAVs), or drones, presents significant security and privacy challenges. Effective drone detection systems are crucial, and deep learning (DL) models have shown great promise. However, the effectiveness of these DL models hinges on vast amounts of diverse, annotated training data. Acquiring such data in the real world is prohibitively expensive, time-consuming, and often hindered by environmental constraints like weather and restricted airspace. While synthetic data offers a cost-effective alternative, existing synthetic datasets for drone detection are limited, with the S-UAV-T dataset being the only publicly available one, and it focuses on UAV-to-UAV detection, deviating from standard surveillance scenarios. This paper aims to bridge this gap by creating a comprehensive synthetic dataset that addresses the limitations of current resources.

Methods

The core of this work lies in the creation of the SynDroneVision dataset. The authors employ an advanced iteration of a data generation pipeline, utilizing Unreal Engine 5.0 and Colosseum (a successor to Microsoft AirSim). Key methodological advancements include:

  • Enhanced Realism: Unreal Engine 5.0's Lumen system provides dynamic global illumination and reflections, significantly improving the realism of rendered scenes compared to previous methods.
  • High-Resolution Capture: To accurately capture intricate lighting details, RGB data is acquired via high-resolution screenshots, overcoming limitations of previous "Scene Capture 2D Actors."
  • Controlled Simulation: The data generation process involves strategically placing virtual camera sensors and defining drone flight trajectories based on probabilistic waypoint selection within the camera's field of view.
  • Parameter Variation: To ensure diversity, the authors systematically vary:
  • Environments: A range of commercially licensed and freely available 3D environments are used, mirroring real-world settings with varying complexity.
  • Drone Models: A selection of widely deployed drone models from commercial asset packs are rendered with realistic textures.
  • Illumination: The dynamic Sun and Sky Actor and Post Process Volume in Unreal Engine are used to simulate diverse lighting conditions, from dawn to dusk and clear to overcast skies.
  • Post-Processing: A subset of images undergoes post-capture blurring using average or Gaussian kernels to further increase data diversity and simulate varying focus conditions.

Datasets

  • SynDroneVision (New Dataset): This paper introduces SynDroneVision, a synthetic dataset specifically designed for RGB-based drone detection. It comprises 131,307 annotated images across 72 sequences, featuring diverse environments, drone models, and lighting conditions. The dataset is split into training (119,307 images), validation (8,000 images), and test (4,000 images) sets.
  • DUT Anti-UAV [53]: Used for comparative analysis and hybrid training. This is a real-world dataset.
  • UAV-Eagle [5] and Drone Dataset [2]: Used as out-of-distribution datasets for evaluating model robustness. These are real-world datasets.

Key Insights and Ideas

  • Synthetic Data's Power: The study strongly suggests that synthetic data, when generated with sufficient realism and diversity, can be a powerful tool for training robust drone detection models, significantly reducing the reliance on costly real-world data collection.
  • Hybrid Training is Key: While purely synthetic training shows promise, combining SynDroneVision with real-world data (like DUT Anti-UAV) leads to substantial improvements in performance across various metrics, highlighting the benefit of a hybrid approach.
  • Unreal Engine 5.0's Advantage: The use of Unreal Engine 5.0 with its Lumen system for dynamic global illumination and reflections is a critical factor in achieving the high fidelity of the synthetic data.
  • Diversity is Paramount: The dataset's strength lies in its deliberate variation of environments, drone models, and lighting conditions, which directly translates to improved model robustness against different scenarios.
  • Simulation-Reality Gap is Real, But Bridgeable: The paper acknowledges the simulation-reality gap, where models trained solely on synthetic data may underperform compared to those trained on real data. However, the results demonstrate that this gap can be effectively narrowed through hybrid training and that synthetic data alone can still yield promising results.

Key Results

  • Hybrid Training Boost: Models trained on a hybrid dataset of SynDroneVision and DUT Anti-UAV showed significant improvements over models trained solely on real-world data. For instance, mAP increased by up to 4.8 percentage points at an IoU of 0.5 and up to 8 percentage points for averaged IoU thresholds.
  • Robustness on Out-of-Distribution Data: Models trained with SynDroneVision (either alone or in hybrid settings) demonstrated notable robustness on out-of-distribution datasets like UAV-Eagle and Drone Dataset, often achieving performance comparable to or even exceeding models trained solely on real-world data.
  • Synthetic-Only Performance: Even when trained exclusively on SynDroneVision, models achieved promising results, with mAP values exceeding 0.639 at an IoU of 0.5 on the DUT Anti-UAV test set, indicating the dataset's practical value.
  • Architecture Impact: The performance gains from integrating SynDroneVision were particularly amplified in more complex YOLO architectures (e.g., YOLOv8m vs. YOLOv8l), suggesting that the benefits scale with model capacity.
  • Reduced Costs: The use of synthetic data significantly reduces the time and costs associated with real-world data acquisition and annotation.

Limitations and Future Work

  • Simulation-Reality Gap: The paper acknowledges that a gap still exists between simulated and real-world conditions, which can negatively impact detection quality. While hybrid training mitigates this, further research into closing this gap is ongoing.
  • Camouflage and Occlusions: Model effectiveness is impaired in scenarios with limited drone visibility due to camouflage effects and occlusions.
  • Future Work: The authors suggest that SynDroneVision can serve as a valuable resource for enriching real-world datasets and improving model robustness. Further exploration into optimal ratios of synthetic to real data for training and investigating the dataset's utility for other tasks beyond detection (e.g., tracking) are potential future directions.

Summary updated

BibTeX
@inproceedings{lenhard2025syndronevision,
  author = {Tamara R. Lenhard and Andreas Weinmann and Kai Franke and Tobias Koch},
  title = {{SynDroneVision: A Synthetic Dataset for Image-Based Drone Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure