Abstract
Tracking small objects like swimmers and boats from a moving drone is notoriously difficult because visual features are often too weak for reliable re-identification. This paper introduces MG-MOT (Metadata-Guided Multi-Object Tracking), which uses a drone’s onboard sensors—GPS, altitude, and gimbal angles—to project 2D image detections into 3D world coordinates. This approach enables robust long-term tracking even when targets temporarily disappear or the camera moves violently. MG-MOT achieved state-of-the-art results, securing 1st place in the 2024 MaCVi UAV-based Maritime Object Tracking Challenge.
Introduction
Multi-Object Tracking (MOT) in maritime environments using Unmanned Aerial Vehicles (UAVs) faces a unique set of challenges. Swimmers and small boats often appear as tiny clusters of pixels, leading to significant visual ambiguity where most targets look indistinguishable. Furthermore, erratic camera motion from drone maneuvers or gimbal adjustments breaks standard overlap-based tracking. Traditional appearance-based Re-Identification (ReID) often fails in these scenarios because there isn't enough visual diversity to differentiate between similar-looking targets over long periods of occlusion.
Methodology
The core innovation of MG-MOT is shifting the tracking logic from the 2D image plane to the 3D world.
3D Projection via Metadata
Instead of relying solely on image-space coordinates, the system utilizes the drone's metadata—including altitude, pitch, yaw, and GPS—to calculate the precise location of an object on the sea surface. By assuming the sea is a flat plane ($z=0$), the framework "un-projects" 2D pixels into 3D world coordinates.
Tracking and Re-Identification
- Short-Term ReID: Frame-to-frame tracking is handled by the Hungarian algorithm, matching detections based on 3D distance. This provides much greater stability than 2D distance during camera tilts or zooms.
- Long-Term ReID: To manage objects that disappear for extended periods, the authors introduce Bi-directional Movement Extrapolation (BiME) to predict target positions based on 3D velocity. This is complemented by Matching Threshold Expansion (MTE), which increases the search area as the duration of an object's absence grows.
- Detection: A modified YOLOv8-x model with a specialized "p2 head" is used to detect the tiny objects characteristic of high-altitude maritime footage.
Experimental Setup
The framework was evaluated using the SeaDroneSee-MOT benchmark, a dataset comprising 21 clips (over 54,000 frames) of humans and boats in open water. This benchmark specifically tests long-term tracking capabilities, requiring systems to re-identify objects after they leave and re-enter the frame.
Key Insights and Results
The MG-MOT approach demonstrates that in environments where visual features are unreliable, geometry and physics offer a more robust alternative. By integrating the drone's own sense of position and orientation, the world is effectively "stabilized," simplifying tracking to a matter of 3D proximity.
MG-MOT achieved 1st place in the 2024 MaCVi UAV-based Maritime Object Tracking Challenge with the following metrics:
- HOTA (Higher Order Tracking Accuracy): 69.5%
- IDF1 (Identification F1-score): 85.9%
The system showed a significant reduction in track fragmentation compared to baselines like BoT-SORT, particularly during rapid gimbal movements.
Conclusion
MG-MOT proves that metadata-guided 3D projection is a powerful tool for UAV-based tracking in feature-poor environments. While the system currently relies on the accuracy of onboard sensors and a "flat earth" assumption, it sets a new standard for maritime MOT. Future work will explore using visual cues to auto-correct metadata synchronization errors and adapting the model for more complex sea states.
Summary updated