Abstract
Visual tracking is a fundamental computer vision task challenged by illumination changes, occlusions, and background clutter. No single tracking algorithm excels in all conditions. This paper introduces MRAT (Multi-Modality Tracker Aggregation), a framework that treats diverse tracking algorithms as "rankers" and fuses their outputs via a probabilistic ranking model. By allowing robust generative models to supervise a discriminative classifier, MRAT effectively mitigates "tracker drift" and achieves superior accuracy across a wide range of challenging video sequences.
Introduction: The Challenge of Tracker Drift
Visual tracking involves maintaining the identity and location of an object throughout a video stream. Traditional trackers generally fall into two categories:
- Generative Models: These learn a descriptive representation of the target (e.g., color histograms). While effective at describing the object, they often struggle to distinguish it from similar-looking backgrounds.
- Discriminative Models: These frame tracking as a binary classification problem (target vs. background). Although powerful, they are prone to self-training drift: small errors in localization are used as positive training samples, causing the tracker to gradually lose the target over time.
MRAT addresses these issues by proposing an ensemble approach that combines multiple trackers using different features (modalities) without relying on heuristic score comparisons.
Methods: Tracking as Rank Aggregation
The core innovation of MRAT is conceptualizing tracking as a Rank Aggregation problem. Instead of raw scores, the system uses the relative rankings of candidate patches.
The process follows these steps:
- Trackers as Rankers: A particle filter generates a set of candidate image patches. Each tracker in the ensemble evaluates these patches and produces a ranked list.
- The Mallows Model: An extended Mallows model—a statistical tool for estimating a "true" ranking from noisy observations—is used to fuse these lists.
- Expertise Weighting: The model dynamically calculates an "expertise" parameter ($\alpha$) for each tracker. Trackers whose top choices align with the final aggregated result gain more influence over time.
- Top-k Optimization: To maintain real-time performance, aggregation is performed only on the "Top-k" candidates.
- Ensemble Supervision: The ensemble combines three generative trackers (Covariance-based, Incremental Subspace, and Spatial-Color GMM) with one discriminative tracker (Multiple Instance Learning, or MIL). The aggregated result is used to update the MIL tracker, allowing generative models to "correct" the classifier before drift occurs.
Datasets and Evaluation
The authors evaluated MRAT on eight challenging video sequences, each highlighting specific tracking difficulties:
- Car & Boat: Low contrast and cluttered backgrounds.
- Small Target: Distractions from nearby moving objects.
- Surf & Shake: Rapid illumination shifts and significant appearance changes.
- Football, Woman, & Girl: Severe occlusions and target-background similarity.
Key Insights
- Ranking as a Universal Language: Converting diverse tracker outputs (probabilities, distances, etc.) into rankings provides a normalized way to combine them.
- Generative Supervision: Using the aggregate rank to select training samples for the MIL tracker solves the self-training problem by providing a "sanity check."
- Temporal Consistency: An EM algorithm with a "warm start" ensures that tracker expertise parameters evolve smoothly across frames.
Key Results
MRAT was compared against five state-of-the-art trackers: MIL, OBT, VDT, LRST, and CMIL.
As shown in the table above, MRAT achieved the lowest Root Mean Square Error (RMSE) in 6 out of the 8 test sequences. Notably, in the "Car" sequence, MRAT's RMSE of 3.8 significantly outperformed the next best tracker (VDT at 10.7). Qualitatively, MRAT demonstrated exceptional robustness in the "Shake" and "Surf" sequences, where other trackers typically failed due to massive appearance changes.
Limitations and Future Work
Despite its robustness, MRAT faces some challenges:
- Computational Overhead: Running multiple trackers and an iterative EM algorithm is demanding. Further optimization is needed for high-resolution, real-time applications.
- Static Ensemble: The current model uses a fixed set of trackers. Future iterations could explore dynamic tracker selection based on environmental context.
- Global Failure: If all underlying trackers fail simultaneously, aggregation cannot recover the target. Integrating a "re-detection" modality could address this.
Summary updated