Multi-Modality Tracker Aggregation: From Generative to Discriminative

Xiaoqin Zhang, Wei Li, Mingyu Fan, Di Wang, Xiuzi YeIJCAI 2015

Abstract

Visual tracking is a fundamental computer vision task challenged by illumination changes, occlusions, and background clutter. No single tracking algorithm excels in all conditions. This paper introduces MRAT (Multi-Modality Tracker Aggregation), a framework that treats diverse tracking algorithms as "rankers" and fuses their outputs via a probabilistic ranking model. By allowing robust generative models to supervise a discriminative classifier, MRAT effectively mitigates "tracker drift" and achieves superior accuracy across a wide range of challenging video sequences.

Introduction: The Challenge of Tracker Drift

Visual tracking involves maintaining the identity and location of an object throughout a video stream. Traditional trackers generally fall into two categories:

  1. Generative Models: These learn a descriptive representation of the target (e.g., color histograms). While effective at describing the object, they often struggle to distinguish it from similar-looking backgrounds.
  2. Discriminative Models: These frame tracking as a binary classification problem (target vs. background). Although powerful, they are prone to self-training drift: small errors in localization are used as positive training samples, causing the tracker to gradually lose the target over time.

MRAT addresses these issues by proposing an ensemble approach that combines multiple trackers using different features (modalities) without relying on heuristic score comparisons.

Methods: Tracking as Rank Aggregation

The core innovation of MRAT is conceptualizing tracking as a Rank Aggregation problem. Instead of raw scores, the system uses the relative rankings of candidate patches.

Overview of the MRAT framework, illustrating the flow from candidate patch generation to rank aggregation and final tracking results.

The process follows these steps:

  1. Trackers as Rankers: A particle filter generates a set of candidate image patches. Each tracker in the ensemble evaluates these patches and produces a ranked list.
  2. The Mallows Model: An extended Mallows model—a statistical tool for estimating a "true" ranking from noisy observations—is used to fuse these lists.
  3. Expertise Weighting: The model dynamically calculates an "expertise" parameter ($\alpha$) for each tracker. Trackers whose top choices align with the final aggregated result gain more influence over time.
  4. Top-k Optimization: To maintain real-time performance, aggregation is performed only on the "Top-k" candidates.
  5. Ensemble Supervision: The ensemble combines three generative trackers (Covariance-based, Incremental Subspace, and Spatial-Color GMM) with one discriminative tracker (Multiple Instance Learning, or MIL). The aggregated result is used to update the MIL tracker, allowing generative models to "correct" the classifier before drift occurs.

Datasets and Evaluation

The authors evaluated MRAT on eight challenging video sequences, each highlighting specific tracking difficulties:

  • Car & Boat: Low contrast and cluttered backgrounds.
  • Small Target: Distractions from nearby moving objects.
  • Surf & Shake: Rapid illumination shifts and significant appearance changes.
  • Football, Woman, & Girl: Severe occlusions and target-background similarity.

Key Insights

  • Ranking as a Universal Language: Converting diverse tracker outputs (probabilities, distances, etc.) into rankings provides a normalized way to combine them.
  • Generative Supervision: Using the aggregate rank to select training samples for the MIL tracker solves the self-training problem by providing a "sanity check."
  • Temporal Consistency: An EM algorithm with a "warm start" ensures that tracker expertise parameters evolve smoothly across frames.

Key Results

MRAT was compared against five state-of-the-art trackers: MIL, OBT, VDT, LRST, and CMIL.

Quantitative comparison of MRAT against other state-of-the-art trackers using Root Mean Square Error (RMSE). Lower values indicate better performance.

As shown in the table above, MRAT achieved the lowest Root Mean Square Error (RMSE) in 6 out of the 8 test sequences. Notably, in the "Car" sequence, MRAT's RMSE of 3.8 significantly outperformed the next best tracker (VDT at 10.7). Qualitatively, MRAT demonstrated exceptional robustness in the "Shake" and "Surf" sequences, where other trackers typically failed due to massive appearance changes.

Limitations and Future Work

Despite its robustness, MRAT faces some challenges:

  • Computational Overhead: Running multiple trackers and an iterative EM algorithm is demanding. Further optimization is needed for high-resolution, real-time applications.
  • Static Ensemble: The current model uses a fixed set of trackers. Future iterations could explore dynamic tracker selection based on environmental context.
  • Global Failure: If all underlying trackers fail simultaneously, aggregation cannot recover the target. Integrating a "re-detection" modality could address this.

Summary updated

BibTeX
@inproceedings{zhang2015multi,
  author = {Xiaoqin Zhang and Wei Li and Mingyu Fan and Di Wang and Xiuzi Ye},
  title = {{Multi-Modality Tracker Aggregation: From Generative to Discriminative}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure