TL;DR
This paper introduces OFTrack, a novel multi-object tracking (MOT) approach that processes sequences of frames ("sections") to improve long-term tracking robustness. By adapting optical flow to "object flow" with enhanced perception and temporal-spatial attention, OFTrack achieves state-of-the-art performance on challenging benchmarks, particularly in scenarios with long intervals between detections.
What Problem Does This Paper Address?
Multi-object tracking (MOT) is crucial for applications like autonomous driving and surveillance, but it's notoriously difficult due to challenges like occlusions, similar-looking objects, and complex motion patterns. Existing methods often process frames individually and rely on motion predictors that struggle with long-term associations, especially when objects are only detected infrequently. This leads to fragmented trajectories and lost identities. The paper aims to bridge this gap by developing a tracker that can effectively reason about object motion over extended periods, even with sparse detections.
Methods
OFTrack introduces a "section-based" approach, processing multiple consecutive frames simultaneously. Its core innovation lies in the Object Flow Tracker (OFTrack), which enhances traditional optical flow to "object flow." This is achieved through two main components:
- Object Perception: This module focuses on understanding the object itself. It uses "object-aware sampling" to focus on relevant features around detected objects and "scale-aware correlation" to handle variations in object size.
- Motion Estimation: This component employs a specialized "temporal-spatial attention" mechanism. It models correlations between objects within a frame (spatial attention) and tracks the same object across multiple frames in the section (temporal attention). This allows for more robust association over longer durations.
To further improve robustness, OFTrack incorporates Temporal Coherent Enhancement:
- Bidirectional Pre-train: The model is pre-trained using masked trajectories, forcing it to learn both forward and backward motion estimation, enhancing generalization.
- Curvature Constraint: A smoothing constraint is applied to trajectory curves to prevent oscillations and ensure smoother, more realistic motion paths.
OFTrack integrates this flow head with a standard object detector (YOLOX) in an end-to-end fashion.
Datasets
The paper evaluates OFTrack on the following datasets:
- MOT17: A standard benchmark for multi-pedestrian tracking.
- MOT20: Another benchmark for multi-pedestrian tracking, known for its crowded scenes.
- DanceTrack: A dataset characterized by highly non-linear motion and frequent object crossovers.
- KITTI: A benchmark for autonomous driving scenarios, including car and pedestrian tracking.
Key Insights and Ideas
- Processing "Sections" is Key: Treating a sequence of frames as a unit, rather than individual frames, significantly improves the ability to handle long-term associations and sparse detections.
- Object Flow vs. Optical Flow: Adapting optical flow to "object flow" by incorporating object semantics and awareness is crucial for robust tracking, moving beyond pixel-level motion.
- Temporal-Spatial Attention for Long-Term Reasoning: The combination of temporal and spatial attention effectively models inter-object and intra-object dynamics across frames, enabling better tracking in complex scenarios.
- Pre-training and Constraints for Robustness: Bidirectional pre-training and curvature constraints are vital for stabilizing motion estimation and preventing trajectory oscillations, especially in long videos.
Key Results
OFTrack achieves state-of-the-art performance across several challenging benchmarks:
- DanceTrack: Achieves an HOTA score of 60.9, further boosted to 63.4 with a ReID module.
- MOT17: Achieves an HOTA score of 63.1.
- MOT20: Achieves an HOTA score of 61.9.
- KITTI: Demonstrates strong performance, with an HOTA score of 73.75 for car tracking and 47.97 for pedestrian tracking.
Ablation studies confirm the effectiveness of each component, with object perception and temporal coherent enhancement showing significant improvements.
Limitations and Future Work
The authors acknowledge that OFTrack currently treats objects as rigid entities and does not explicitly handle non-rigid deformations. This limitation can lead to tracking drift when object aspect ratios change significantly.
Future work will focus on integrating deformation estimation counterparts into the OFTrack framework to enhance its adaptability to a broader range of scenarios with non-rigid objects.
Summary updated