Robust Visual Tracking via Inverse Nonnegative Matrix Factorization

Fanghui Liu, Tao Zhou, Keren Fu, Irene Yu‐Hua Gu, Jie Yang

TL;DR

This paper introduces "Inverse Nonnegative Matrix Factorization" (Inverse NMF) for robust visual tracking. By treating the problem in reverse, it effectively uses both foreground and background information to learn a more discriminative appearance model, leading to improved tracking accuracy, especially in challenging scenarios like occlusions and motion blur.

What Problem Does This Paper Address?

Visual tracking aims to follow a target object through a sequence of images. A critical challenge is maintaining an accurate "appearance model" of the target over time, which can change due to variations in lighting, pose, and occlusions. Traditional methods often struggle when the target's appearance deviates significantly from its initial representation. While Nonnegative Matrix Factorization (NMF) has been applied to this problem, existing NMF-based trackers often focus solely on the target's appearance (foreground) and neglect valuable information from the background, limiting their robustness.

Methods

The core innovation is the Inverse Nonnegative Matrix Factorization (Inverse NMF). Unlike conventional NMF which decomposes a data matrix into basis and coefficient matrices (X ≈ UV), Inverse NMF flips this: it aims to represent the basis matrix (U) as a linear combination of candidate patches (Y) with corresponding coefficient vectors (C), i.e., U ≈ YC.

Here's a breakdown of their approach:

  • Initialization: A simple tracker (like MeanShift) is used to collect initial target (foreground) and background patches.
  • Basis Learning (Inverse NMF): The key idea is to learn a basis matrix (U) that can represent both foreground and background candidates. This is achieved by minimizing the reconstruction error between the basis matrix and candidate patches, while also imposing a local coordinate constraint. This constraint encourages the coefficient vectors (C) to be sparse and locally similar, meaning similar candidate patches should have similar coefficient representations.
  • Feature Extraction: The learned coefficient vectors (C) from Inverse NMF are treated as discriminative features. These vectors capture the essence of how a candidate patch can be represented by the learned basis.
  • Classification: A Support Vector Machine (SVM) classifier is trained using these coefficient vectors to distinguish between the target (positive samples) and the background (negative samples).
  • Online Updating: To adapt to appearance changes, the algorithm employs incremental learning. The appearance models (both foreground and background) are updated periodically using new samples from the sequence. This involves updating the basis matrix (U) and coefficient matrix (V) incrementally.
  • Two-Stage Optimal State Screening: To further enhance robustness, after identifying potential target candidates, the algorithm uses a generative approach (minimizing reconstruction error) to select the most likely target location from the top candidates, rather than just picking the one with the highest confidence from the classifier.

Datasets

The paper evaluates their method on several publicly available video sequences. Specific datasets mentioned include:

  • DavidOutdoor
  • Jumping
  • Deer
  • Caviar1
  • Caviar3
  • Carscale
  • Occlusion1
  • Basketball
  • Human7
  • Car4

The paper does not introduce any new datasets.

Key Insights and Ideas

  • Reversing NMF for Robustness: The most significant insight is that by reversing the NMF formulation (U ≈ YC instead of X ≈ UV), the method can effectively learn a basis that captures both target and background characteristics. This allows for a more discriminative separation.
  • Leveraging Background Information: Explicitly incorporating background information into the appearance model is crucial. This helps the tracker distinguish the target even when its appearance changes, as the background provides a stable reference.
  • Local Coordinate Constraint for Sparsity and Similarity: The imposed local coordinate constraint is key to generating meaningful and discriminative coefficient vectors. It ensures that similar visual patterns are represented similarly, while also promoting sparsity, which aids in feature extraction.
  • Generative-Discriminative Hybrid: The two-stage screening process, combining the discriminative power of SVM with the reconstruction error minimization of generative methods, offers a robust way to select the final target location.

Key Results

The proposed Inverse NMF tracker demonstrates superior performance compared to seven state-of-the-art methods across various challenging scenarios.

  • Quantitative:
    • Achieved the best or second-best performance in terms of Center Location Error (CLE) on several sequences, notably outperforming others in "Jumping" and "Deer" with an average CLE of 7.3 pixels.
  • Similarly, it achieved the best or second-best overlap rates, indicating accurate bounding box localization, with an average overlap rate of 0.74.
  • Qualitative:
    • Showed remarkable robustness against heavy occlusions, pose changes, and illumination variations.
  • Outperformed other trackers in scenarios with fast motion and motion blur, where abrupt changes make accurate prediction difficult.
  • The paper highlights specific examples where other trackers drift or fail completely, while the proposed method maintains tracking.

Limitations and Future Work

  • Limitations:
    • The paper mentions that the negative template set is updated every 5 frames, which might be a trade-off between computation and model fitness. A more frequent or adaptive update strategy could potentially improve performance further.
  • While robust, the method's performance might still be challenged by extremely rapid and unpredictable appearance changes or complete target disappearance for extended periods.
  • Future Work:
    • The authors suggest that further exploration of the local coordinate constraint and its impact on the coefficient vectors could lead to even more discriminative features.
  • Investigating more advanced online updating strategies for the appearance models could enhance adaptability.
  • Extending the Inverse NMF framework to other computer vision tasks beyond tracking could be a promising direction.

Summary updated

Abstract

The establishment of robust target appearance model over time is an overriding concern in visual tracking. In this paper, we propose an inverse nonnegative matrix factorization (NMF) method for robust appearance modeling. Rather than using a linear combination of nonnegative basis matrices for each target image patch in the conventional NMF, the proposed method is a reverse thought to conventional NMF tracker. It utilizes both the foreground and background information, and imposes a local coordinate constraint, where the basis matrix is sparse matrix from the linear combination of candidates with corresponding nonnegative coefficient vectors. Inverse NMF is used as a feature encoder, where the resulting coefficient vectors are fed into a SVM classifier for separating the target from the background. The proposed method is tested on several videos and compared with seven state-of-the-art methods. Our results have provided further support to the effectiveness and robustness of the proposed method.
BibTeX
@inproceedings{liu2016robust,
  author = {Fanghui Liu and Tao Zhou and Keren Fu and Irene Yu‐Hua Gu and Jie Yang},
  title = {{Robust Visual Tracking via Inverse Nonnegative Matrix Factorization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2016},
  doi = {10.1109/icassp.2016.7471925},
}
Figure