Multi-Frame Attention with Feature-Level Warping for Drone Crowd Tracking

Takanori Asanomi, Kazuya Nishimura, Ryoma BiseWACV 2023 · OpenReview

Abstract

Drone crowd tracking is essential for applications like crowd management and surveillance. However, the small size of objects and point-level annotations lack discriminative features, making standard similarity-based tracking difficult. We propose a multi-frame attention mechanism with feature-level warping to aggregate temporal context. Feature-level warping aligns features across frames, while multi-frame attention aggregates these aligned features. Our method outperforms state-of-the-art techniques on the DroneCrowd dataset, particularly in tracking consistency.

Introduction

Tracking individuals in dense crowds from aerial drone footage presents unique challenges compared to standard multi-object tracking (MOT). In typical MOT, objects are large enough for bounding boxes to capture rich visual features. In contrast, drone surveillance often relies on point-level annotations because individuals appear as tiny, similar-looking blobs. Furthermore, drone movement creates unstable backgrounds, and lighting variations lead to inconsistent detections. To address these issues, we leverage temporal context from local areas to improve tracking robustness.

Methodology

Our framework utilizes a Siamese U-net architecture enhanced by a novel temporal aggregation module consisting of three primary steps:

Feature-Level Warping

To compare features across time, they must be spatially aligned. Given the motion of both the drone and the subjects, we estimate "feature-level flow" to warp features from adjacent frames into the coordinate system of the reference frame. This ensures that subsequent attention mechanisms operate on features corresponding to the same physical locations.

Multi-Frame Attention

Once aligned, a self-attention mechanism aggregates information across $M$ frames. By weighting clearer features more heavily (e.g., when a subject is obscured by a shadow in one frame but clear in another), the model constructs a more robust representation. We employ a linear complexity formulation to ensure efficiency, avoiding the quadratic scaling issues of standard Vision Transformers.

Backward Warping

Aggregated features are warped back to their original positions to ensure the final output heatmap remains spatially accurate for each frame in the sequence.

Dataset and Evaluation

We evaluated our approach on the DroneCrowd dataset, which contains 112 video sequences and over 4.8 million head point annotations. The dataset features diverse conditions, including varying illumination (Sunny, Cloudy, Night), object scales, and crowd densities.

Visualization of tracking consistency and dataset diversity.

Results

The proposed method was compared against the previous state-of-the-art, STNNet. Our model achieved a Localization Mean Average Precision (L-mAP) of 43.43, compared to STNNet's 40.45. More significantly, our method showed a substantial improvement in Tracking mAP (T-mAP), reaching 42.08—a nearly 10-point increase over STNNet's 32.50. This indicates a marked reduction in identity switching and false negatives.

Conclusion and Future Work

While our multi-frame attention significantly improves tracking in most conditions, night scenes remain a challenge due to extremely weak visual signals. Future research will explore better feature representations for low-light environments and the integration of motion priors to assist the warping module when visual flow is difficult to estimate.

Code is available at: https://github.com/asanomitakanori/mfa-feature-warping

Summary updated

Abstract

Drone crowd tracking has various applications such as crowd management and video surveillance. Unlike in general multi-object tracking, the size of the objects to be tracked are small, and the ground truth is given by a point-level annotation, which has no region information. This causes the lack of discriminative features for finding the same objects from many similar objects. Thus, similarity-based tracking techniques, which are widely used for multi-object tracking with bounding-box, are difficult to use. To deal with this problem, we take into account the temporal context of the local area. To aggregate temporal context in a local area, we propose a multi-frame attention with feature-level warping. The feature-level warping can align the features of the same object in multiple frames, and then multi-frame attention can effectively aggregate the temporal context from the warped features. The experimental results show the effectiveness of our method. Our method outperformed the state-of-the-art method in DroneCrowd dataset. The code is publicly available in https://github.com/asanomitakanori/mfa-feature-warping.
BibTeX
@inproceedings{asanomi2023multi,
  author = {Takanori Asanomi and Kazuya Nishimura and Ryoma Bise},
  title = {{Multi-Frame Attention with Feature-Level Warping for Drone Crowd Tracking}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=lXuTn5axg2},
}
Figure