Tracking Persons-of-Interest via Unsupervised Representation Adaptation

Shun Zhang, Jia-Bin Huang, Jongwoo Lim, Yihong Gong, Jinjun Wang, Narendra Ahuja, Ming-Hsuan Yang

Abstract

Tracking individuals in unconstrained videos, such as movies or music videos, is challenging due to frequent shot changes, varying lighting, and rapid motion. This paper presents an unsupervised representation adaptation framework that fine-tunes a face-recognition Convolutional Neural Network (CNN) to a specific video's visual domain. By leveraging spatio-temporal and contextual constraints (e.g., clothing similarity) as "free" supervision, the system adapts to the unique characteristics of the footage. A novel Symmetric Triplet Loss (SymTriplet) is introduced to enhance feature discriminability. Evaluations on TV sitcoms, music videos, and pedestrian datasets demonstrate significant improvements over state-of-the-art trackers.

Introduction

Multi-target tracking (MTT) is relatively straightforward when a camera is stationary and motion is continuous. However, "unconstrained" videos—such as TV sitcoms or YouTube music videos—present significant challenges for traditional trackers. These videos feature frequent shot changes, rapid camera movements, and dramatic changes in a character's appearance due to makeup, lighting, or viewing angles.

Standard trackers often rely on low-level features (like color histograms) that aren't discriminative enough to tell two similar-looking actors apart across different scenes. While deep learning (CNNs) can provide better features, a model trained on a general dataset often fails to generalize to the specific, idiosyncratic visual domain of a particular movie or music video.

Methodology

The authors propose a four-stage pipeline that moves from general knowledge to video-specific expertise:

  1. Supervised Pre-training: The process begins with a standard AlexNet architecture pre-trained on the CASIA-WebFace dataset to learn basic identity-preserving features.
  2. Discovering Training Samples: This is the "unsupervised" core. The system analyzes the target video to find labels without human help. It uses spatio-temporal constraints (faces in the same continuous tracklet are the same person; two faces in the same frame must be different people) and contextual constraints (using clothing/torso similarity to link people across different shots).
  3. Representation Adaptation: The CNN is fine-tuned using these discovered samples. The authors introduce a novel Symmetric Triplet Loss (SymTriplet). Unlike standard triplet loss, which only pushes a "negative" sample away from one "anchor," SymTriplet pushes the negative sample away from both positive samples in a triplet simultaneously, creating a more stable and discriminative embedding space.
  4. Hierarchical Linking: Finally, Hierarchical Agglomerative Clustering (HAC) is used to link short tracklets into long-term trajectories across the entire video.
The proposed pipeline for unsupervised representation adaptation and hierarchical linking.

Datasets and Evaluation

The study evaluates the method on three distinct types of video:

  • TV Sitcoms: The Big Bang Theory (BBT) and Buffy the Vampire Slayer. These are challenging due to dim lighting and frequent scene changes.
  • Music Videos: A new benchmark dataset consisting of 8 YouTube music videos. These feature rapid cuts, heavy makeup changes, and high-motion dancing.
  • Pedestrians: The DukeMTMC dataset for tracking people across multiple surveillance cameras.

Key Results

The proposed method significantly outperformed state-of-the-art trackers (like mTLD and ADMM) across all datasets:

  • Clustering Purity: On the Big Bang Theory dataset, the adapted features achieved a weighted purity of up to 0.95, compared to just 0.37 for traditional HOG features.
  • Tracking Accuracy: In music videos, the "Ours-SymTriplet-Contx" model achieved a 64.1% MOTA (Multi-Object Tracking Accuracy), whereas the baseline pre-trained model only reached 51.5%.
  • Identity Switches: The method drastically reduced "ID switches," the primary failure mode in across-shot tracking.
Example of tracking output on challenging music video footage showing input frames and resulting face detections.

Conclusion and Future Work

The core insight of this work is that the structure of a video provides its own supervision. By incorporating contextual cues like clothing, the system bridges gaps between shots where faces may be obscured. While effective, the approach currently requires significant computational time (approx. one hour per video) and depends heavily on initial face detection accuracy. Future research may explore multi-modal information, such as audio or script alignment, to provide even stronger unsupervised constraints.

Summary updated

BibTeX
@article{zhang2020tracking,
  author = {Shun Zhang and Jia-Bin Huang and Jongwoo Lim and Yihong Gong and Jinjun Wang and Narendra Ahuja and Ming-Hsuan Yang},
  title = {{Tracking Persons-of-Interest via Unsupervised Representation Adaptation}},
  journal = {Int. J. Comput. Vis.},
  year = {2020},
  doi = {10.1007/s11263-019-01212-1},
}
Figure