Object Tracking With Structured Metric Learning

Xiaolin Zhao, Zhuofan Xu, Boxin Zhao, Xiaolong Chen, Zongzhe LiIEEE Access 2019 · DOI

Introduction

Object tracking in computer vision is a fundamental challenge typically approached through either generative or discriminative frameworks. Generative models focus on matching a target template, often ignoring background information, while discriminative models treat tracking as a binary classification problem (target vs. background). However, these approaches often fail to capture the complex spatial nuances of video frames or adapt to dynamic appearance changes caused by lighting, rotation, and occlusion.

This paper introduces Structured Metric Learning (SML), a novel framework that bridges the gap between structured learning and adaptive distance metrics. By incorporating spatial constraints directly into the metric learning process, SML achieves state-of-the-art robustness against common tracking challenges like background clutter and target deformation.

The SML Framework

The SML framework operates through three integrated phases designed to maintain an accurate and spatially-aware representation of the target.

Structured Metric Learning

Instead of relying on a fixed distance measure like Euclidean distance, SML learns a custom metric matrix $M$. The "structured" component of this learning process enforces a spatial constraint: samples that are geographically further from the target in the image space should have a larger distance in the feature space than those closer to it.

The SML framework learns a metric matrix M by selecting target and neighbor samples across temporal frames.

A key contribution of this work is the derivation of a closed-form solution for $M$. This allows the metric to be calculated efficiently without the need for iterative optimization, making it suitable for the high-speed requirements of video tracking. The system utilizes Histogram of Oriented Gradients (HOG) as its primary feature descriptor.

Tracking via KNN Distance

During the tracking phase, the system employs a sliding window to generate candidate patches in a new frame. Rather than matching against a single template, the tracker calculates the average distance between these candidates and a "target buffer" containing previous successful tracking results.

The tracking phase involves sampling candidate patches and calculating their distance to a target buffer using the learned metric.

By using a K-Nearest Neighbors (K=10) approach, the system becomes significantly more resilient to outliers and noise compared to single-match methods. The candidate with the minimum average distance to its neighbors in the target buffer is identified as the new target location.

FIFO Update Scheme

To adapt to appearance changes over time, the tracker maintains two buffers—Target and Neighbor—using a First-In-First-Out (FIFO) strategy. As new frames are processed, older samples are replaced by recent ones, and the metric $M$ is recalculated to reflect the target's current state. This ensures the model remains discriminative even as the target undergoes rotation or illumination shifts.

Experimental Evaluation

The SML tracker was evaluated on the OTB-51 benchmark dataset, which includes 51 video sequences featuring challenges such as fast motion, scale variation, and out-of-plane rotation.

Key Results

  • Overall Performance: SML achieved a leading Average Center Location Error (CLE) of 31.2 pixels and a Precision of 0.765, outperforming ten other state-of-the-art trackers including Struck, SCM, and TLD.
  • Deformation Resilience: In sequences involving significant target deformation, SML outperformed the nearest competitor (SCM) by a substantial 20% in precision.
  • Background Clutter: The structured metric proved highly effective at distinguishing the target from complex backgrounds, a scenario where traditional binary classifiers often struggle.

Conclusion and Future Work

SML demonstrates that formulating distance as a spatial proxy allows for a more discriminative feature space specifically tuned to the current scene. While the current implementation shows exceptional robustness, the authors note that the dense neighbor sampling results in a computational complexity of $O(mnd)$. Future work will focus on optimizing the algorithm for faster execution and exploring further theoretical justifications for its performance in extreme conditions involving simultaneous illumination shifts and complex deformations.

Summary updated

BibTeX
@article{zhao2019object,
  author = {Xiaolin Zhao and Zhuofan Xu and Boxin Zhao and Xiaolong Chen and Zongzhe Li},
  title = {{Object Tracking With Structured Metric Learning}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2950690},
}
Figure