Representation Alignment Contrastive Regularization for Multi-Object Tracking

Zhonglin Liu, Shujie Chen, Jianfeng Dong, Xun Wang, Di ZhouarXiv 2024 · DOI · OpenReview

Abstract

Multi-Object Tracking (MOT) relies heavily on effective data association to link detections across video frames. Traditional methods often struggle with erratic motion or require complex, computationally expensive deep learning architectures. This paper presents the Representation Alignment Module (RAM), a lightweight, single-layer transformer designed to align object features using intuitive spatio-temporal rules. Trained with contrastive regularization, RAM serves as a plug-and-play enhancement for existing trackers. Experimental results on MOT17, MOT20, and BDD100K benchmarks demonstrate that RAM significantly improves tracking accuracy and identity preservation with negligible impact on processing speed.

Introduction

The "Tracking-by-Detection" (TBD) paradigm is the standard approach for MOT, involving object detection followed by temporal linking. However, modeling the spatio-temporal relationships for data association remains a challenge. Existing solutions typically fall into two categories: rule-based methods (like Kalman filters) which are interpretable but brittle, and deep learning methods which are powerful but often opaque and difficult to integrate. The proposed Representation Alignment Module (RAM) bridges this gap by providing a simple, interpretable, and high-performing module that can be easily integrated into various tracking workflows.

Methodology

The core innovation is the Representation Alignment Module (RAM), a detector-free component consisting of a single-layer transformer encoder. RAM processes bounding box coordinates or pre-extracted features rather than raw video frames, ensuring high efficiency.

The module is trained using Contrastive Regularization governed by two primary rules:

  1. Temporal Rule: Ensures that representations of the same object in consecutive frames are similar, while representations of different objects are pushed apart.
  2. Spatial Rule: Aligns representations of different parts of the same object (e.g., a "mark" box vs. the full body) to maintain spatial consistency.
The RAM architecture and its integration into the tracking-by-detection pipeline.

Once trained, RAM produces "aligned" features used to generate a new affinity matrix. This matrix is combined with the original tracker's affinity matrix via a weighted sum, allowing for seamless integration with state-of-the-art trackers like ByteTrack or OC-SORT.

Experimental Results

The effectiveness of RAM was evaluated on major benchmarks including MOT17, MOT20, and BDD100K. The researchers integrated RAM into five leading trackers: JDE, CSTrack, TransTrack, ByteTrack, and OC-SORT.

Quantitative results showing the performance boost provided by RAM variants across multiple base trackers.

Key findings include:

  • Accuracy Improvements: On the MOT17 validation set, adding the spatio-temporal RAM (STRAM) to ByteTrack increased the IDF1 score from 79.07 to 80.87.
  • Enhanced Stability: Identity Switches (IDS) were consistently reduced. For JDE, IDS dropped from 473 to 374.
  • Efficiency: The computational overhead is minimal. For instance, FairMOT's frame rate only decreased slightly from 22.5 FPS to 21.1 FPS.
  • Noise Robustness: RAM acts as a denoiser, learning to ignore mismatches from traditional assignment algorithms by training on all triplets.

Conclusion

The Representation Alignment Module offers a robust and efficient way to enhance multi-object tracking. By enforcing simple spatio-temporal rules through contrastive regularization, it provides interpretable feature alignment that translates to significant performance gains. Future work may explore more complex fusion techniques and address edge cases like extreme speeds or identical appearances in crowded scenes.

Check out the code: The authors have made their implementation available on GitHub at https://github.com/liuzhonglincc/RATracker.

Summary updated

Abstract

Achieving high-performance in multi-object tracking algorithms heavily relies on modeling spatio-temporal relationships during the data association stage. Mainstream approaches encompass rule-based and deep learning-based methods for spatio-temporal relationship modeling. While the former relies on physical motion laws, offering wider applicability but yielding suboptimal results for complex object movements, the latter, though achieving high-performance, lacks interpretability and involves complex module designs. This work aims to simplify deep learning-based spatio-temporal relationship models and introduce interpretability into features for data association. Specifically, a lightweight single-layer transformer encoder is utilized to model spatio-temporal relationships. To make features more interpretative, two contrastive regularization losses based on representation alignment are proposed, derived from spatio-temporal consistency rules. By applying weighted summation to affinity matrices, the aligned features can seamlessly integrate into the data association stage of the original tracking workflow. Experimental results showcase that our model enhances the majority of existing tracking networks' performance without excessive complexity, with minimal increase in training overhead and nearly negligible computational and storage costs.
BibTeX
@misc{liu2024representation,
  author = {Zhonglin Liu and Shujie Chen and Jianfeng Dong and Xun Wang and Di Zhou},
  title = {{Representation Alignment Contrastive Regularization for Multi-Object Tracking}},
  year = {2024},
  eprint = {2404.02562},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.02562},
}
Figure