Abstract
Drone crowd tracking is essential for applications like crowd management and surveillance. However, the small size of objects and point-level annotations lack discriminative features, making standard similarity-based tracking difficult. We propose a multi-frame attention mechanism with feature-level warping to aggregate temporal context. Feature-level warping aligns features across frames, while multi-frame attention aggregates these aligned features. Our method outperforms state-of-the-art techniques on the DroneCrowd dataset, particularly in tracking consistency.
Introduction
Tracking individuals in dense crowds from aerial drone footage presents unique challenges compared to standard multi-object tracking (MOT). In typical MOT, objects are large enough for bounding boxes to capture rich visual features. In contrast, drone surveillance often relies on point-level annotations because individuals appear as tiny, similar-looking blobs. Furthermore, drone movement creates unstable backgrounds, and lighting variations lead to inconsistent detections. To address these issues, we leverage temporal context from local areas to improve tracking robustness.
Methodology
Our framework utilizes a Siamese U-net architecture enhanced by a novel temporal aggregation module consisting of three primary steps:
Feature-Level Warping
To compare features across time, they must be spatially aligned. Given the motion of both the drone and the subjects, we estimate "feature-level flow" to warp features from adjacent frames into the coordinate system of the reference frame. This ensures that subsequent attention mechanisms operate on features corresponding to the same physical locations.
Multi-Frame Attention
Once aligned, a self-attention mechanism aggregates information across $M$ frames. By weighting clearer features more heavily (e.g., when a subject is obscured by a shadow in one frame but clear in another), the model constructs a more robust representation. We employ a linear complexity formulation to ensure efficiency, avoiding the quadratic scaling issues of standard Vision Transformers.
Backward Warping
Aggregated features are warped back to their original positions to ensure the final output heatmap remains spatially accurate for each frame in the sequence.
Dataset and Evaluation
We evaluated our approach on the DroneCrowd dataset, which contains 112 video sequences and over 4.8 million head point annotations. The dataset features diverse conditions, including varying illumination (Sunny, Cloudy, Night), object scales, and crowd densities.
Results
The proposed method was compared against the previous state-of-the-art, STNNet. Our model achieved a Localization Mean Average Precision (L-mAP) of 43.43, compared to STNNet's 40.45. More significantly, our method showed a substantial improvement in Tracking mAP (T-mAP), reaching 42.08—a nearly 10-point increase over STNNet's 32.50. This indicates a marked reduction in identity switching and false negatives.
Conclusion and Future Work
While our multi-frame attention significantly improves tracking in most conditions, night scenes remain a challenge due to extremely weak visual signals. Future research will explore better feature representations for low-light environments and the integration of motion priors to assist the warping module when visual flow is difficult to estimate.
Code is available at: https://github.com/asanomitakanori/mfa-feature-warping
Summary updated