MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Model

Changcheng Xiao, Qiong Cao, Zhigang Luo, Long Lan

Introduction

Multiple Object Tracking (MOT) is a fundamental task in computer vision, primarily dominated by the "tracking-by-detection" paradigm. In this framework, objects are detected in individual frames and subsequently associated across time to form trajectories. Traditionally, the Kalman Filter (KF) has served as the backbone for motion prediction in these systems. However, the Kalman Filter relies on a linear motion assumption that frequently fails in real-world scenarios characterized by erratic, non-linear movements, such as professional dancing or high-intensity sports.

Furthermore, many MOT systems rely heavily on appearance-based Re-Identification (Re-ID) to resolve ambiguities. In environments like DanceTrack or SportsMOT, where subjects often wear identical uniforms and undergo frequent occlusions, appearance cues become unreliable. MambaTrack addresses these challenges by introducing a learnable motion predictor based on the Mamba State Space Model (SSM), providing a robust baseline for motion-centric tracking.

The MambaTrack Framework

MambaTrack replaces the traditional linear motion model with a sophisticated sequence modeling architecture. The framework consists of two primary innovations integrated into a standard detection-association pipeline.

Mamba moTion Predictor (MTP)

The core of the system is the Mamba moTion Predictor (MTP). Unlike standard State Space Models, MTP utilizes a bi-directional Mamba encoding layer. By processing historical bounding box coordinates in both forward and backward temporal directions, the model captures a more holistic representation of motion patterns. The MTP is trained to predict the "offset"—the change in position and scale—for the subsequent frame, allowing it to adapt to complex, non-linear trajectories that would baffle a standard Kalman Filter.

Tracklet Patching Module (TPM)

To maintain trajectory continuity during periods of occlusion, the authors developed the Tracklet Patching Module (TPM). When a detection is missing in a specific frame, the system operates in an autoregressive mode: the MTP uses its own previous predictions as input to generate the next predicted state. This mechanism "patches" gaps in the trajectory, keeping the tracklet alive and making it significantly easier to re-associate the object once it reappears.

Experimental Results

MambaTrack was evaluated on two challenging benchmarks: DanceTrack, which features fluid, non-linear motion, and SportsMOT, which involves rapid movements and frequent overlaps in basketball, football, and volleyball.

State-of-the-Art Performance

  • DanceTrack: MambaTrack achieved a HOTA (Higher Order Tracking Accuracy) of 56.8, surpassing the previous state-of-the-art motion tracker, OC_SORT, by 2.2 percentage points. It also reached a peak IDF1 of 57.8, demonstrating superior consistency in maintaining object identities over time.
  • SportsMOT: The model achieved a HOTA of 72.6, a significant improvement of nearly 10 percentage points over ByteTrack, which relies on a standard Kalman Filter.

Efficiency and Complexity

One of the primary advantages of the Mamba architecture is its linear complexity relative to sequence length. While Transformers offer powerful modeling capabilities, their quadratic complexity can be prohibitive for real-time tracking. MambaTrack provides comparable modeling power with much higher efficiency. The tracking component (excluding detection) requires only 11.37 ms per frame, accounting for approximately 19% of the total processing pipeline latency.

Key Insights

  1. The Bi-Directional Advantage: Ablation studies revealed that the Bi-Mamba block is critical for performance. Moving from a vanilla unidirectional Mamba to a bi-directional block resulted in a 2.5% increase in HOTA, suggesting that temporal context from both directions is vital for understanding complex motion.
  2. Motion as a Primary Signal: The results suggest that in scenarios where appearance is indistinguishable (e.g., athletes in uniform), a robust, learnable motion model is a more effective signal for association than traditional Re-ID features.
  3. Scalability: Mamba's efficiency allows for longer "look-back" windows than LSTMs or Transformers, enabling the model to learn more nuanced long-term motion dependencies without a massive computational overhead.

Conclusion and Future Work

MambaTrack establishes a new, simple baseline for motion-based multiple object tracking. By replacing the rigid assumptions of the Kalman Filter with a flexible, bi-directional State Space Model, it achieves significant gains in tracking accuracy for complex scenarios.

Future research directions include reducing the dependency on the external object detector's quality and exploring the integration of Mamba-based motion features with end-to-end Transformer architectures. Additionally, further tuning of the temporal window size for specific sports could yield even higher precision in specialized tracking environments.

Summary updated

BibTeX
@inproceedings{xiao2024mambatrack,
  author = {Changcheng Xiao and Qiong Cao and Zhigang Luo and Long Lan},
  title = {{MambaTrack: A Simple Baseline for Multiple Object Tracking with State Space Model}},
  booktitle = {ACM Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=b6VJWTQKLF},
}
Figure