FOLT: Fast Multiple Object Tracking from UAV-captured Videos Based on Optical Flow

Mufeng Yao, Jiaqi Wang, Jinlong Peng, Mingmin Chi, Chao Liu

TL;DR

This paper introduces FOLT, a novel Multiple Object Tracking (MOT) system designed for challenging UAV-captured videos. FOLT leverages optical flow for both feature augmentation and motion prediction, significantly improving tracking accuracy for small, fast-moving, and irregularly moving objects, outperforming existing state-of-the-art methods on challenging UAV datasets.

What Problem Does This Paper Address?

Multiple Object Tracking (MOT) is a crucial task in computer vision, with applications ranging from autonomous driving to surveillance. However, applying MOT to videos captured by Unmanned Aerial Vehicles (UAVs) presents unique challenges. These include:

  • Small Object Size: Objects appear much smaller in aerial views, making them harder to detect and track.
  • Blurred Appearance: Fast motion of both the ground objects and the UAV platform can lead to motion blur, further degrading object appearance and making traditional appearance-based tracking methods unreliable.
  • Large and Irregular Motion: Objects in UAV footage often exhibit rapid and unpredictable movements, making it difficult for trackers to maintain object identities across frames.

Existing MOT methods, often relying on appearance features, struggle with these issues. This paper aims to bridge this gap by developing a robust and efficient MOT system specifically for UAV scenarios.

Methods

FOLT tackles the challenges of UAV-based MOT through two primary novel components, both leveraging optical flow:

  1. Flow-Guided Feature Augmentation (FGFA): This technique enhances object detection features. It uses optical flow to "sample" features from the previous frame at the current frame's spatial locations. These sampled features are then fused with the current frame's features in an attentive manner. This process helps to "sharpen" the appearance of small or blurred objects by incorporating information from their past states, guided by their motion.
  2. Flow-Guided Motion Prediction (FGMP): Instead of relying solely on traditional motion models like the Kalman Filter (KF), which struggle with rapid and irregular motion, FGMP uses optical flow to predict an object's future position. Specifically, it extracts optical flow information around the object's bounding box and feeds it into a lightweight motion prediction network (MPN). This allows the tracker to better anticipate object movements, especially in scenarios with large displacements between frames.

The overall FOLT architecture integrates these components with a modern object detector (YOLOX) and a fast optical flow estimator (FastFlowNet). The process involves:

  • Extracting detection features and optical flow.
  • Augmenting detection features using FGFA.
  • Performing object detection with the augmented features.
  • Predicting object motion using FGMP.
  • Associating detected objects with predicted positions to form tracks.

Datasets

The authors evaluated FOLT on two widely used UAV-based MOT datasets:

  • Visdrone [52]: This dataset features multi-class object tracking from UAV perspectives, known for its small object sizes and complex motion patterns.
  • UAVDT [10]: This dataset specifically focuses on car tracking in aerial views, encompassing various urban and road scenarios with significant motion.

The paper uses the standard train-test splits provided by these datasets.

Key Insights and Ideas

  • Optical Flow as a Dual-Purpose Tool: The paper demonstrates that optical flow is not just for motion estimation but can be effectively used to enhance object detection features (FGFA) and improve motion prediction (FGMP) in challenging scenarios.
  • Appearance is Overrated for UAV MOT: The authors' findings suggest that for UAV-based MOT, where objects are small and often blurred, relying heavily on appearance features is less effective than robust motion modeling. Their method, which primarily uses motion cues, outperforms methods that incorporate appearance.
  • Bridging the Detection-Tracking Gap: FGFA directly addresses the challenge of detecting small and blurred objects by leveraging temporal information through optical flow, effectively improving the input to the detection head.
  • Beyond Kalman Filters for Motion: FGMP offers a more adaptive and accurate way to predict object trajectories in the presence of large and irregular motions, outperforming traditional KF-based methods in these scenarios.

Key Results

FOLT achieved state-of-the-art performance on both the Visdrone and UAVDT datasets.

  • Visdrone: FOLT achieved a MOTA of 42.1% and an IDF1 of 56.9%, outperforming the previous best method OCSORT by 2.5% in MOTA and UAVMOT by 5.9% in IDF1.
  • UAVDT: FOLT achieved a MOTA of 48.5% and an IDF1 of 68.3%, also surpassing previous bests.
  • Speed: FOLT achieved a speed of 29.4 FPS, which is faster than several competing methods, demonstrating its efficiency.
  • Ablation Studies: Both FGFA and FGMP individually contributed to performance improvements. Their combination yielded the best results, validating their effectiveness. FGFA was particularly effective in improving tracking of small objects, while FGMP showed significant gains in scenarios with high Mean Relative Acceleration (MRA).

Limitations and Future Work

  • Computational Cost: While FOLT is designed to be fast, integrating optical flow estimation still adds computational overhead compared to methods that don't use it. The paper notes that FastFlowNet was chosen for its balance of accuracy and speed.
  • Generalizability to Other Domains: The paper focuses on UAV-captured videos. While the principles might extend, further validation on other challenging MOT domains (e.g., crowded urban scenes with different object scales) would be beneficial.
  • Future Work: The authors suggest that the combination of FGFA and FGMP leads to the best performance, implying that further exploration of how these components can be more deeply integrated or optimized could yield even better results. The paper also highlights the potential for further speed improvements.

Summary updated

Abstract

Multiple object tracking (MOT) has been successfully investigated in computer vision. However, MOT for the videos captured by unmanned aerial vehicles (UAV) is still challenging due to small object size, blurred object appearance, and very large and/or irregular motion in both ground objects and UAV platforms. In this paper, we propose FOLT to mitigate these problems and reach fast and accurate MOT in UAV view. Aiming at speed-accuracy trade-off, FOLT adopts a modern detector and light-weight optical flow extractor to extract object detection features and motion features at a minimum cost. Given the extracted flow, the flow-guided feature augmentation is designed to augment the object detection feature based on its optical flow, which improves the detection of small objects. Then the flow-guided motion prediction is also proposed to predict the object's position in the next frame, which improves the tracking performance of objects with very large displacements between adjacent frames. Finally, the tracker matches the detected objects and predicted objects using a spatially matching scheme to generate tracks for every object. Experiments on Visdrone and UAVDT datasets show that our proposed model can successfully track small objects with large and irregular motion and outperform existing state-of-the-art methods in UAV-MOT tasks.
BibTeX
@inproceedings{yao2023folt,
  author = {Mufeng Yao and Jiaqi Wang and Jinlong Peng and Mingmin Chi and Chao Liu},
  title = {{FOLT: Fast Multiple Object Tracking from UAV-captured Videos Based on Optical Flow}},
  booktitle = {ACM Multimedia},
  year = {2023},
}
Figure