Multi-Object Tracking Meets Moving UAV

Shuai Liu, Xin Li, Huchuan Lu, You He

Introduction

Multi-Object Tracking (MOT) is a foundational technology for autonomous systems, typically optimized for stable, ground-level perspectives such as surveillance or dashcam footage. However, transitioning these algorithms to Unmanned Aerial Vehicles (UAVs) introduces significant complexities. Drones operate in 3D space with non-linear motion, leading to three primary challenges: irregular camera movement (rotation and tilt), tiny target resolution due to high altitudes, and extreme data imbalance where common objects like cars vastly outnumber rare categories like vans or bicycles.

Key challenges in UAV-based tracking: imbalanced categories, rapid view changes, and irregular camera motion.

The UAVMOT Framework

To address these hurdles, the authors introduce UAVMOT, a tracking-by-detection framework specifically engineered for the chaotic dynamics of aerial video. The architecture integrates three novel components designed to maintain track continuity despite camera instability.

The UAVMOT architecture featuring the ID Feature Update (IDFU) and Adaptive Motion Filter (AMF).

Core Components

  • ID Feature Update (IDFU) Module: Unlike traditional trackers that rely on static appearance templates, IDFU utilizes a correlation layer to compare features across adjacent frames. This allows the object's "identity" to evolve adaptively as the drone's viewing angle shifts.
  • Adaptive Motion Filter (AMF): This module acts as an intelligent switch. Under smooth flight conditions, it employs standard association. When abnormal motion (jerks or rotations) is detected, it activates a Local Relation Filter. This filter tracks the relative geometry—distances and angles—between an object and its neighbors. Because these relative spatial relationships remain invariant even if the camera rotates, the tracker avoids losing targets during sudden maneuvers.
  • Gradient Balanced Focal (GBF) Loss: To combat the "long-tail" distribution of aerial datasets, the GBF loss prioritizes smaller bounding boxes and rare classes. By scaling loss based on the inverse of an object's area and implementing equalization loss, the model is forced to learn features for tiny and infrequent targets effectively.

Key Insights and Methodology

The central innovation of UAVMOT lies in the realization that relative geometry is invariant. While a drone's rotation disrupts absolute coordinate tracking, the spatial relationship between adjacent objects remains constant. By shifting the focus from absolute positioning to relational filtering, the system gains immunity to camera shake.

Furthermore, the framework treats identity as a dynamic target. In UAV footage, an object's appearance can change drastically in seconds as the drone flies over it. The IDFU module ensures the tracker's internal representation of an object stays current with its visual reality.

Results and Performance

UAVMOT was rigorously tested on the VisDrone2019 and UAVDT benchmarks, which are known for their extreme class imbalance and small object density.

  • VisDrone2019: The framework achieved a 36.1% MOTA (Multi-Object Tracking Accuracy), significantly outperforming previous state-of-the-art methods like MOTR and TrackFormer. Notably, it reached an IDF1 of 51.0%, indicating superior consistency in maintaining object IDs.
  • UAVDT: In vehicle-centric urban environments, it achieved 46.4% MOTA, demonstrating high robustness.

Qualitative analysis confirms that while standard trackers like FairMOT often fail during sudden drone rotations, UAVMOT maintains steady tracks by leveraging its relational motion model.

Conclusion and Future Work

While UAVMOT sets a new standard for aerial tracking accuracy, its current processing speed of 12 FPS on a DLA34 backbone limits its use for real-time, on-board deployment. Future research will focus on optimizing the model's parameters to reach the 30 FPS threshold required for live UAV operations without compromising the accuracy provided by its relational filtering approach.

Summary updated

Abstract

Multi-object tracking in unmanned aerial vehicle (UAV) videos is an important vision task and can be applied in a wide range of applications. However, conventional multi-object trackers do not work well on UAV videos due to the challenging factors of irregular motion caused by moving camera and view change in 3D directions. In this paper, we propose a UAVMOT network specially for multi-object tracking in UAV views. The UAVMOT introduces an ID feature update module to enhance the object's feature association. To better handle the complex motions under UAV views, we develop an adaptive motion filter module. In addition, a gradient balanced focal loss is used to tackle the imbalance categories and small objects detection problem. Experimental results on the VisDrone2019 and UAVDT datasets demonstrate that the proposed UAVMOT achieves considerable improvement against the state-of-the-art tracking methods on UAV videos.
BibTeX
@inproceedings{liu2022multi,
  author = {Shuai Liu and Xin Li and Huchuan Lu and You He},
  title = {{Multi-Object Tracking Meets Moving UAV}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00867},
}
Figure