Abstract
In multi-object tracking (MOT), maintaining accurate appearance and motion information for each object is crucial for matching tracks with new detections. Traditional methods often model targets in isolation, leading to confusion when objects look similar. We propose a novel multi-track pooling module that allows for the simultaneous consideration of all tracks during memory updates. This approach, combined with a training strategy that generates hard tracking episodes, creates a highly discriminative appearance model. Our framework enables the use of simple, real-time greedy data association while achieving state-of-the-art performance on public MOT benchmarks.
The Challenge of Isolated Tracking
Multi-Object Tracking aims to link detections across video frames to form continuous trajectories. A significant hurdle in this process is ambiguity: if two individuals in a scene appear similar—such as two people wearing identical suits—a tracker focusing only on a single person's history can easily lose track or switch identities.
Historically, researchers have addressed this by employing complex data association algorithms like the Hungarian algorithm or Multiple Hypothesis Tracking. While effective, these methods are computationally intensive and often require "offline" processing—looking at future frames to resolve past ambiguities—which prevents real-time application. The goal of this work is to build an appearance model so powerful that these complex optimizations become unnecessary.
Multi-Track Pooling: A Joint Approach
To overcome these limitations, we introduce a multi-track pooling module. Unlike standard approaches that update each track's memory independently, our model considers the entire scene context.
For any specific track, the system aggregates a "max-pooled" summary of the memories of all other tracks currently in the scene. This provides immediate "online negative examples," allowing the model to distinguish a target not just by its own features, but by how it differs from its immediate neighbors. This discriminative power is captured within a Bilinear LSTM (BLSTM) framework, which excels at modeling the multiplicative interactions between a track's memory and new detections.
Efficiency through Greedy Association
Because the multi-track pooling module makes the appearance model so robust, we can bypass complex global optimization. Instead, we use a simple greedy data association strategy: matching a track to the detection with the highest likelihood and moving on. This significantly reduces computational overhead, allowing the tracker to run at 20–24 Hz, meeting the requirement for true real-time processing.
Experimental Results
We evaluated our approach (BLSTM-MTP) on standard benchmarks including MOT16, MOT17, and MOT19. The results demonstrate that joint modeling of tracks directly leads to more stable trajectories and fewer errors.
As shown in the table above, our method significantly improves the IDF1 score (a metric for identity preservation) and substantially reduces Identity Switches (IDS) and Fragmentations (Frag) compared to the baseline. On the MOT17 challenge, our online tracker achieved a competitive 60.5 IDF1, outperforming many complex offline association schemes.
Conclusion
By shifting the focus from complex post-processing to building a more powerful, context-aware appearance model, we demonstrate that real-time, high-accuracy multi-object tracking is achievable. The multi-track pooling module effectively leverages scene-wide information to resolve local ambiguities, proving that in tracking, context is indeed king.
Summary updated