TransVisDrone: Spatio-Temporal Transformer for Vision-based Drone-to-Drone Detection in Aerial Videos

Tushar Sangam, Ishan Rajendrakumar Dave, Waqas Sultani, Mubarak Shah

TL;DR

TransVisDrone is a novel, end-to-end spatio-temporal transformer framework designed specifically for detecting drones from the perspective of another drone. By combining a spatial backbone with VideoSwin transformer blocks, the method achieves state-of-the-art accuracy on challenging datasets while maintaining the high throughput necessary for real-time deployment on edge devices like the NVIDIA Jetson.

What Problem Does This Paper Address?

Drone-to-drone (D2D) detection is a critical capability for collision avoidance, autonomous flight coordination, and security (e.g., intercepting rogue drones). However, it is significantly harder than standard object detection for three main reasons:

  1. Extreme Scale: Target drones often occupy as little as 0.07% of the total frame area—essentially a few pixels.
  2. Erratic Motion: Both the target and the source drone are moving, leading to motion blur and unpredictable trajectories.
  3. Complex Backgrounds: Drones must be spotted against cluttered landscapes, clouds, or abrupt illumination changes (like flying toward the sun).

Previous state-of-the-art methods, such as DogFight, relied on complex, multi-stage pipelines involving frame patching, manual tracking, and non-differentiable components. These are difficult to train and often too slow for the real-time constraints of edge computing.

Methods

The authors propose TransVisDrone, a simplified, fully differentiable architecture that processes video clips rather than isolated frames.

  • Spatial Feature Extraction: The model uses CSPDarkNet-53 (the backbone of YOLOv5) to extract multi-scale spatial features ($P_3, P_4, P_5$). This allows the model to capture details at different resolutions, which is vital for "tiny" objects.
  • Spatio-Temporal Transformer: The core innovation is the integration of VideoSwin Transformer blocks. Unlike standard transformers that use global self-attention (which is computationally ruinous for video), VideoSwin uses shifted 3D windows. This allows the model to learn temporal dependencies—how a drone moves across frames—without a massive increase in compute.
  • Temporally-Consistent Augmentation (TCA): To train the model effectively, the authors apply the same random augmentations (like color jitter or perspective shifts) to every frame in a sampled clip. This ensures that the model learns actual object motion rather than artifacts introduced by inconsistent data processing.
  • End-to-End Detection: The final features are passed through a standard detection head and Non-Maximum Suppression (NMS) to produce bounding box sequences.

Datasets

The model was evaluated on three major benchmarks:

  • NPS-Drones: 50 videos of custom delta-wing drones captured with GoPros.
  • FL-Drones: A smaller dataset featuring a mix of indoor and outdoor drone footage.
  • Airborne Object Tracking (AOT): A large-scale, high-resolution dataset from Amazon Prime Air containing over 5.9 million frames. This dataset is particularly challenging due to its diverse airborne objects (planes, birds, helicopters) and high-altitude perspectives.

Key Insights and Ideas

  • Temporal Context is the "Aha!" Moment: The authors found that single-frame detectors often fail because a tiny drone is indistinguishable from background noise or a sensor artifact. By looking at a short temporal window (3–5 frames), the model can distinguish a drone by its consistent trajectory and motion patterns.
  • Efficiency via Windowing: By using the VideoSwin architecture, the team achieved the benefits of transformers (long-range dependency modeling) while keeping the frame rate high enough for real-world use.
  • The Resolution Trade-off: One of the most practical findings was that reducing resolution from 1280p to 640p resulted in only a 3% drop in Average Precision (AP) but a massive 250% increase in throughput, enabling real-time performance on edge hardware.

Key Results

TransVisDrone sets a new bar for D2D detection performance:

  • Accuracy: Achieved 0.95 AP on NPS and 0.80 AP on the AOT dataset, outperforming the previous best method (DogFight) by 6% on AOT.
  • Speed: On a high-end NVIDIA RTX A6000, it reaches 24.6 FPS. More importantly, on an NVIDIA Jetson Xavier NX (a common edge board for drones), it achieves a real-time 33 FPS at 640p resolution.
  • Safety Metrics: The model successfully detected drones in 46% of "closed encounter" flights (defined as continuous detection for at least 3 seconds within a 300m range), a significant step toward reliable autonomous collision avoidance.

Limitations and Future Work

While TransVisDrone is a leap forward, the authors note a few areas for growth:

  • Tracking vs. Detection: The current model focuses on detection. While it processes clips, it doesn't explicitly assign unique IDs to multiple drones over long durations (multi-object tracking).
  • Monocular Distance Estimation: The paper suggests that future iterations could extend the spatio-temporal features to estimate the distance of the target drone using only the single camera feed, which would be invaluable for navigation.
  • False Positives: While the False Positive Per Image (FPPI) is low, in high-stakes aviation, even a few false alarms can trigger unnecessary evasive maneuvers. Refinement in extremely cluttered "below-the-horizon" scenarios remains a goal.

Summary updated

Abstract

Drone-to-drone detection using visual feed has crucial applications, such as detecting drone collisions, detecting drone attacks, or coordinating flight with other drones. However, existing methods are computationally costly, follow non-end-to-end optimization, and have complex multi-stage pipelines, making them less suitable for real-time deployment on edge devices. In this work, we propose a simple yet effective framework, TransVisDrone, that provides an end-to-end solution with higher computational efficiency. We utilize CSPDarkNet-53 network to learn object-related spatial features and VideoSwin model to improve drone detection in challenging scenarios by learning spatio-temporal dependencies of drone motion. Our method achieves state-of-the-art performance on three challenging real-world datasets (Average [email protected]): NPS 0.95, FLDrones 0.75, and AOT 0.80, and a higher throughput than previous methods. We also demonstrate its deployment capability on edge devices and its usefulness in detecting drone-collision (encounter). Project: https://tusharsangam.github.io/TransVisDrone-project-page/
BibTeX
@inproceedings{sangam2023transvisdrone,
  author = {Tushar Sangam and Ishan Rajendrakumar Dave and Waqas Sultani and Mubarak Shah},
  title = {{TransVisDrone: Spatio-Temporal Transformer for Vision-based Drone-to-Drone Detection in Aerial Videos}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=0luSoXu468},
}

Loading related content...

Figure