Graph Networks for Multiple Object Tracking

Jiahe Li, Xu Gao, Tingting Jiang

TL;DR

This paper introduces a novel "near-online" Multiple Object Tracking (MOT) framework that leverages end-to-end Graph Networks to model global relationships between targets. By iteratively updating nodes, edges, and a shared global variable, the system moves beyond traditional static graph models to better handle occlusions and complex object interactions, setting new performance standards on the MOT16 and MOT17 benchmarks.

What Problem Does This Paper Address?

Multiple Object Tracking (MOT) is the "who is where" problem in computer vision: identifying and following multiple targets (usually pedestrians) across a video sequence. While modern detectors are good at finding objects in a single frame, associating those objects across time remains difficult due to:

  1. Similar Appearances: Two people wearing the same color clothes.
  2. Occlusions: One person walking behind a tree or another person.
  3. Local vs. Global Context: Most existing methods only look at "local" pairs (e.g., "Does detection A in frame 10 match object B in frame 9?"). They ignore the "global" context—how the movement or appearance of all objects in the scene informs the identity of a single one.

Traditional graph-based MOT methods often use static graphs, where the features of the nodes (objects) and edges (connections) are fixed. This paper argues that graphs should be dynamic and iterative, allowing the model to "reason" about the scene as it processes information.

Methods

The authors propose a 4-step end-to-end graph network architecture. Unlike standard graph models, their approach treats tracking as an inference process where the graph components evolve.

Overview of the proposed MOT framework pipeline, detailing the stages from feature extraction to data association and missing detection handling.

1. Dual Graph Networks

The system uses two separate graph networks to process different "cues":

  • Appearance Graph Network (AGN): Uses a ResNet-34 backbone to extract visual features. It learns to calculate similarity based on how people look.
  • Motion Graph Network (MGN): Uses 2D coordinates and velocity vectors to predict where an object should be.

2. The 4-Step Update Mechanism

Each graph block performs four sequential updates:

  • Edge Update I: Calculates initial association scores between objects and new detections.
  • Node Update: Refines the features of the object nodes based on the detections they are likely connected to.
  • Global Update: Updates a "Global Variable"—a feature vector that represents the state of the entire scene (e.g., crowd density or camera motion).
  • Edge Update II: Produces the final association scores, now informed by the updated nodes and the global context.

3. Handling Missing Detections

To fix "holes" left by imperfect detectors, the authors include two fallback strategies:

  • Single Object Tracker (SOT): If a target isn't detected, a dedicated tracker tries to find it in the current frame based on its last known position.
  • Detection Recovery: For targets lost for longer periods, a linear motion model predicts their path until they reappear.

Datasets

The model was evaluated on the two most prominent benchmarks in the field:

  • MOT16: Contains 14 video sequences with varying crowd densities and camera angles.
  • MOT17: An extension of MOT16 that provides more accurate ground truth and tests models against three different object detectors (DPM, Faster R-CNN, and SDP).

Key Insights and Ideas

Conceptual illustration of the graph network utilizing a global variable to model scene-wide context during association. Source, p. 1

Conceptual illustration of the graph network utilizing a global variable to model scene-wide context during association.
  • The Power of the Global Variable: One of the most significant "aha" moments is the effectiveness of the global variable. By encoding scene-wide information, the model can resolve ambiguities that local features can't. For example, if the camera moves suddenly, the global variable captures this shift, helping the motion model realize that all objects moved, not just one.
  • Iterative Refinement: By updating node features during the association process, the model can "clean up" noisy appearance data. If an object is partially occluded, the graph can use information from the edges to maintain a stable identity.
  • Complementary Strengths: The ablation studies show that motion and appearance are highly complementary. Appearance is best for distinguishing unique individuals, while motion is the "anchor" when people look similar or walk close together.

Key Results

The proposed method achieved state-of-the-art results among "near-online" trackers:

  • MOT16: Achieved a MOTA (MOT Accuracy) of 47.7 using public detections and 58.4 using private detections.
  • MOT17: Achieved a MOTA of 50.2, significantly outperforming previous near-online methods like MHT_bLSTM.
  • Identity Stability: The model showed a marked improvement in IDF1 (Identity F1-score), indicating it is much better at maintaining long-term tracks without "switching" IDs between people.

Limitations and Future Work

  • Near-Online Latency: Because the model requires a small temporal window to resolve associations, it isn't strictly "real-time" in the sense of zero-latency processing, though it is close.
  • Initialization Sensitivity: The motion-based components (MGN) are somewhat sensitive to how the neural network weights are initialized, which can lead to variance in performance.
  • Future Directions: The authors suggest expanding the graph to include more historical frames and explicitly modeling the physical interactions between objects within the graph structure itself.

Summary updated

Abstract

Multiple object tracking (MOT) task requires reasoning the states of all targets and associating these targets in a global way. However, existing MOT methods mostly focus on the local relationship among objects and ignore the global relationship. Some methods formulate the MOT problem as a graph optimization problem. However, these methods are based on static graphs, which are seldom updated. To solve these problems, we design a new near-online MOT method with an end-to-end graph network. Specifically, we design an appearance graph network and a motion graph network to capture the appearance and the motion similarity separately. The updating mechanism is carefully designed in our graph network, which means that nodes, edges and the global variable in the graph can be updated. The global variable can capture the global relationship to help tracking. Finally, a strategy to handle missing detections is proposed to remedy the defect of the detectors. Our method is evaluated on both the MOT16 and the MOT17 benchmarks, and experimental results show the encouraging performance of our method.
BibTeX
@inproceedings{li2020graph,
  author = {Jiahe Li and Xu Gao and Tingting Jiang},
  title = {{Graph Networks for Multiple Object Tracking}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  doi = {10.1109/wacv45572.2020.9093347},
}
Figure