Discriminative Appearance Modeling With Multi-Track Pooling for Real-Time Multi-Object Tracking

Chanho Kim, Fuxin Li, Mazen Alotaibi, James M. Rehg

Abstract

In multi-object tracking (MOT), maintaining accurate appearance and motion information for each object is crucial for matching tracks with new detections. Traditional methods often model targets in isolation, leading to confusion when objects look similar. We propose a novel multi-track pooling module that allows for the simultaneous consideration of all tracks during memory updates. This approach, combined with a training strategy that generates hard tracking episodes, creates a highly discriminative appearance model. Our framework enables the use of simple, real-time greedy data association while achieving state-of-the-art performance on public MOT benchmarks.

The Challenge of Isolated Tracking

Multi-Object Tracking aims to link detections across video frames to form continuous trajectories. A significant hurdle in this process is ambiguity: if two individuals in a scene appear similar—such as two people wearing identical suits—a tracker focusing only on a single person's history can easily lose track or switch identities.

Historically, researchers have addressed this by employing complex data association algorithms like the Hungarian algorithm or Multiple Hypothesis Tracking. While effective, these methods are computationally intensive and often require "offline" processing—looking at future frames to resolve past ambiguities—which prevents real-time application. The goal of this work is to build an appearance model so powerful that these complex optimizations become unnecessary.

Multi-Track Pooling: A Joint Approach

To overcome these limitations, we introduce a multi-track pooling module. Unlike standard approaches that update each track's memory independently, our model considers the entire scene context.

The proposed multi-track pooling architecture (left) compared to traditional isolated track modeling (right).

For any specific track, the system aggregates a "max-pooled" summary of the memories of all other tracks currently in the scene. This provides immediate "online negative examples," allowing the model to distinguish a target not just by its own features, but by how it differs from its immediate neighbors. This discriminative power is captured within a Bilinear LSTM (BLSTM) framework, which excels at modeling the multiplicative interactions between a track's memory and new detections.

Efficiency through Greedy Association

Because the multi-track pooling module makes the appearance model so robust, we can bypass complex global optimization. Instead, we use a simple greedy data association strategy: matching a track to the detection with the highest likelihood and moving on. This significantly reduces computational overhead, allowing the tracker to run at 20–24 Hz, meeting the requirement for true real-time processing.

Experimental Results

We evaluated our approach (BLSTM-MTP) on standard benchmarks including MOT16, MOT17, and MOT19. The results demonstrate that joint modeling of tracks directly leads to more stable trajectories and fewer errors.

Ablation study showing the performance gains of our proposed method over the baseline B-LSTM.

As shown in the table above, our method significantly improves the IDF1 score (a metric for identity preservation) and substantially reduces Identity Switches (IDS) and Fragmentations (Frag) compared to the baseline. On the MOT17 challenge, our online tracker achieved a competitive 60.5 IDF1, outperforming many complex offline association schemes.

Conclusion

By shifting the focus from complex post-processing to building a more powerful, context-aware appearance model, we demonstrate that real-time, high-accuracy multi-object tracking is achievable. The multi-track pooling module effectively leverages scene-wide information to resolve local ambiguities, proving that in tracking, context is indeed king.

Summary updated

Abstract

In multi-object tracking, the tracker maintains in its memory the appearance and motion information for each object in the scene. This memory is utilized for finding matches between tracks and detections, and is updated based on the matching. Many approaches model each target in isolation and lack the ability to use all the targets in the scene to jointly update the memory. This can be problematic when there are similarly looking objects in the scene. In this paper, we solve the problem of simultaneously considering all tracks during memory updating, with only a small spatial overhead, via a novel multi-track pooling module. We additionally propose a training strategy adapted to multi-track pooling which generates hard tracking episodes online. We show that the combination of these innovations results in a strong discriminative appearance model under the bilinear LSTM tracking framework, enabling the use of greedy data association to achieve online tracking performance. Our experiments demonstrate real-time, state-of-the-art online tracking performance on public multi-object tracking (MOT) datasets.
BibTeX
@inproceedings{kim2021discriminative,
  author = {Chanho Kim and Fuxin Li and Mazen Alotaibi and James M. Rehg},
  title = {{Discriminative Appearance Modeling With Multi-Track Pooling for Real-Time Multi-Object Tracking}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=rZZMPnCwlc},
}
Figure