Detector-Augmented SAMURAI for Long-Duration Drone Tracking

Tamara R. Lenhard, Anis Kacem, Hichem Snoussi, Tobias Koch

Abstract

Robust long-term tracking of drones is a critical requirement for modern surveillance systems, given their increasing threat potential. While detector-based approaches typically achieve strong frame-level accuracy, they often suffer from temporal inconsistencies caused by frequent detection dropouts. Despite its practical relevance, research on RGB-based drone tracking is still limited and largely reliant on conventional motion models. Meanwhile, foundation models like SAMURAI have established their effectiveness across other domains, exhibiting strong category-agnostic tracking performance. However, their applicability in drone-specific scenarios has not been investigated yet. Motivated by this gap, we present the first systematic evaluation of SAMURAI's potential for robust drone tracking in urban surveillance settings. Furthermore, we introduce a detector-augmented extension of SAMURAI to mitigate sensitivity to bounding-box initialization and sequence length. Our findings demonstrate that the proposed extension significantly improves robustness in complex urban environments, with pronounced benefits in long-duration sequences - especially under drone exit-re-entry events. The incorporation of detector cues yields consistent gains over SAMURAI's zero-shot performance across datasets and metrics, with success rate improvements of up to +0.393 and FNR reductions of up to -0.475.

Introduction

Tracking drones in urban environments presents a unique set of challenges for computer vision. Drones are typically small, move at high velocities, and are often designed with camouflage to blend into complex backgrounds. Current methodologies generally fall into two categories: individual frame detectors (e.g., YOLO) and Visual Object Trackers (VOT). Detectors are accurate but prone to "dropouts"—frames where the target is missed due to motion blur or lighting—resulting in flickering trajectories. Conversely, VOTs provide smoother tracking but are susceptible to "drift" over long sequences, eventually losing the target entirely.

Furthermore, while foundation models like SAMURAI (based on SAM 2) have shown promise in general tracking tasks, their performance on drone-specific data remains largely unexplored. This paper addresses these gaps by introducing a detector-augmented SAMURAI framework designed for robust, long-term drone surveillance.

Methodology

The proposed framework, Detector-Augmented SAMURAI, integrates three core components to ensure stable tracking:

  1. Core Tracker: SAMURAI provides the foundational ability to segment and follow the drone's mask using a Kalman Filter-based motion model and instance-level memory.
  2. Specialized Detector: The system incorporates YOLO-FEDER FusionNet, a state-of-the-art model optimized for detecting drones in cluttered urban scenes.
  3. Prediction Fusion Module: This module acts as the integration layer. It monitors detector confidence and periodically "re-prompts" the tracker with fresh bounding boxes. This mechanism is crucial for correcting drift and handling "exit-re-entry" events, where a drone leaves and subsequently returns to the camera's field of view.
Tracking sequence showing detector dropouts and tracker stability.

Experimental Setup and Datasets

The study evaluates the framework using two primary benchmarks:

  • DUT Anti-UAV: A standard dataset comprising 20 outdoor sequences.
  • Custom Long-Duration Datasets (R1 & R2): To specifically test long-term robustness, the authors introduced four new high-resolution sequences of up to 6,000 frames each. These sequences emphasize exit-re-entry scenarios, which are common failure points for traditional trackers.

Results and Discussion

The integration of a detector into the SAMURAI pipeline yields transformative improvements in tracking performance. Key findings include:

  • Enhanced Success Rates: On long-duration sequences, the augmented model achieved a success rate improvement of up to +0.393 over the baseline SAMURAI.
  • Reduced Missed Detections: The False Negative Rate (FNR) saw a significant reduction, dropping by as much as 0.475.
  • Mitigation of Mask Degradation: Without detector guidance, trackers often drift toward background structures (like tree branches) or fail to capture the full extent of the drone (e.g., missing rotors). Continuous prompting from the detector ensures the bounding box remains centered and correctly scaled.
Alignment of detector and tracker bounding boxes.

The system also maintains high efficiency. The YOLO-FEDER component processes frames in approximately 12.4 ms on an NVIDIA A100, supporting the viability of this approach for real-time surveillance applications.

Conclusion and Future Work

The detector-augmented SAMURAI framework significantly advances the state-of-the-art in long-term drone tracking. However, challenges remain. Tracking "sub-pixel" targets—drones that appear as only a few pixels at extreme distances—remains difficult due to limited visual evidence. Additionally, the fusion logic requires careful parameter tuning (e.g., IoU thresholds) which may vary by environment. Future research will focus on improving performance under heavy occlusion and optimizing parameter sensitivity for diverse surveillance conditions.

Summary updated

Abstract

Robust long-term tracking of drone is a critical requirement for modern surveillance systems, given their increasing threat potential. While detector-based approaches typically achieve strong frame-level accuracy, they often suffer from temporal inconsistencies caused by frequent detection dropouts. Despite its practical relevance, research on RGB-based drone tracking is still limited and largely reliant on conventional motion models. Meanwhile, foundation models like SAMURAI have established their effectiveness across other domains, exhibiting strong category-agnostic tracking performance. However, their applicability in drone-specific scenarios has not been investigated yet. Motivated by this gap, we present the first systematic evaluation of SAMURAI's potential for robust drone tracking in urban surveillance settings. Furthermore, we introduce a detector-augmented extension of SAMURAI to mitigate sensitivity to bounding-box initialization and sequence length. Our findings demonstrate that the proposed extension significantly improves robustness in complex urban environments, with pronounced benefits in long-duration sequences - especially under drone exit-re-entry events. The incorporation of detector cues yields consistent gains over SAMURAI's zero-shot performance across datasets and metrics, with success rate improvements of up to +0.393 and FNR reductions of up to -0.475.
BibTeX
@misc{lenhard2026detector,
  author = {Tamara R. Lenhard and Anis Kacem and Hichem Snoussi and Tobias Koch},
  title = {{Detector-Augmented SAMURAI for Long-Duration Drone Tracking}},
  year = {2026},
  eprint = {2601.04798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.04798},
}
Figure