Abstract
Multi-Object Tracking (MOT) is often hindered by rigid temporal segmentation and a lack of spatial context, resulting in frequent identity switches and fragmented trajectories. This paper introduces a novel framework that leverages Reinforcement Learning (RL) for adaptive key frame extraction and Graph Convolutional Networks (GCNs) for intra-frame feature fusion. By dynamically segmenting video sequences and modeling the spatial dependencies between concurrent detections, the proposed method achieves state-of-the-art performance on the MOT17 benchmark, notably improving HOTA and IDF1 scores while significantly reducing identity switches.
Introduction
The primary goal of Multi-Object Tracking (MOT) is to maintain consistent identities for multiple targets across a video sequence. Despite recent progress, two major hurdles persist:
- Rigid Temporal Processing: Traditional trackers often process video in fixed-length segments, failing to account for the dynamic nature of object visibility and occlusion.
- Spatial Neglect: Many systems focus on temporal appearance consistency but ignore the spatial relationships between neighboring objects. This often leads to ID switches when similar-looking targets cross paths or move in close proximity.
Methodology
The proposed framework addresses these issues through two main components integrated into a hierarchical tracking baseline (SUSHI).
Adaptive Key Frame Extraction (KFE)
Instead of arbitrary video chunking, the KFE module employs Q-learning to adaptively determine optimal segmentation points. The RL agent is rewarded based on feature consistency between the start and end of a segment, allowing the model to "mine" key frames that best capture motion information and bridge the gap between short-term tracking and long-term re-identification.
Intra-Frame Feature Fusion (IFF)
To mitigate the effects of occlusion, the IFF module utilizes a Graph Convolutional Network (GCN). By treating each detection in a frame as a node in a graph, the GCN enables information flow between neighboring objects. This "global context" allows the model to maintain a robust signature for partially hidden targets by leveraging the features of visible surrounding objects.
Experimental Results
The model was evaluated on the MOT17 benchmark using private detections. The results demonstrate significant improvements over existing state-of-the-art trackers such as ByteTrack and FairMOT.
Performance Metrics
- HOTA: 68.6 (vs. 67.0 for the SUSHI baseline)
- IDF1: 81.0, reflecting high identity consistency.
- AssA: 66.6, indicating superior association accuracy.
- ID Switches (IDS): Reduced to 893, a substantial improvement over ByteTrack (2196) and FairMOT (3303).
Discussion and Conclusion
The integration of adaptive temporal segmentation and spatial context modeling provides a more robust solution for MOT. While the framework achieves state-of-the-art accuracy, future work should address the computational overhead introduced by RL and GCN components. Additionally, the modular nature of the KFE component suggests potential for integration into other tracking architectures to enhance their temporal reasoning capabilities.
Summary updated