The current state of face tracking for video surveillance

Chris Whiten, Wassim Bouachir, Guillaume-Alexandre Bilodeau, Robert LaganièreR-libre 2012

Abstract

This paper provides a comprehensive survey and evaluation of modern face and person tracking algorithms specifically tailored for video surveillance. By categorizing methods into appearance and motion models and applying a Technology Readiness Level (TRL) framework, the authors identify sparse representation-based methods as the current "state-of-the-art" for handling the occlusions and appearance changes typical of high-security environments like airports.

Introduction

While general object tracking is a well-studied field in computer vision, video surveillance introduces a unique set of "non-negotiable" requirements. A surveillance tracker must operate in real-time, handle frequent occlusions (people walking behind pillars or each other), manage drastic lighting changes, and—most importantly—maintain a consistent identity for a target even after they leave and re-enter the camera's field of view.

The authors address the gap between theoretical tracking research and practical deployment. They evaluate whether current algorithms are actually ready for specific real-world scenarios: isolated people in lanes (PIL), controlled chokepoints (security gates), uncontrolled doorways, and dense outdoor crowds.

Methodology: Appearance and Motion Models

The tracking problem is decomposed into two primary components: appearance models and motion models.

Appearance Models

Appearance models define how the tracker "sees" and identifies the target. The paper categorizes these into several types:

  • Template Matching: Simple but prone to "drift" when a person turns their head.
  • Histograms: Robust to motion blur but lose spatial information. Fragtrack is highlighted for using part-based histograms to handle partial occlusions.
  • Local Features: Keypoints like SIFT, SURF, or binary descriptors (BRIEF/BRISK) are excellent for handling rotation and scale changes.
  • Sparse Representations: A key focus of the paper. These methods use a dictionary of templates and solve an $\ell_1$-minimization problem to reconstruct the target, effectively treating occlusions as "noise."
  • Learned Models: Real-time machine learning (e.g., Adaboost, Multiple Instance Learning) is used to distinguish the target from the background.
Categorization of tracking algorithms by feature type.

Motion Models

Motion models govern how the tracker moves the search area:

  • Bayesian Filters: Specifically Particle Filters, which represent possible locations as a set of "particles" evolving over time.
  • Tracking-by-Detection: Leveraging object detectors to re-seed or verify tracker results.

Key Insights and Technical Considerations

Handling Occlusion and Drift

A major challenge in tracking is the "drift" dilemma: updating templates too frequently can cause the tracker to learn the background, while updating too rarely leads to failure during pose changes. Sparse representation methods, such as the Collaborative Tracker, mitigate this by representing a face as a linear combination of previous views, allowing the system to mathematically "ignore" occluding objects like books or hands.

Part-based tracking handling partial face occlusion.

Computational Requirements

For particle filters to be viable in surveillance, they require thousands of particles. The paper notes that GPU acceleration is mandatory, enabling the processing of 10,000 particles in under 90ms, which is essential for real-time high-fidelity tracking.

Technology Readiness Level (TRL)

The authors use the TRL (1-9) scale to assess maturity. Most high-end trackers currently sit at TRL 4 (lab validation) or TRL 5 (relevant environment validation). None are yet considered "flight-proven" (TRL 9) for fully autonomous surveillance.

Results and Evaluation

The studied methods were evaluated across standard benchmarks like PETS-2006, CAVIAR, and TUD:

  • Isolated Tracking (PIL): Distribution Fields achieved 100% accuracy on occluded face sequences with a mean error of only 5 pixels.
  • Dense Traffic: In the CAVIAR dataset, the Adaptive Structural Local Sparse Tracker significantly outperformed older methods, reducing tracking error to 2.3 pixels compared to Fragtrack's 116 pixels.
  • Outdoor Performance: The CRF Multi-Target Tracker successfully tracked 70% of trajectories in difficult outdoor settings, though identity switches (IDS) remain a challenge.

Conclusion and Future Work

Despite advancements, significant gaps remain. There is a lack of research and datasets for "chokepoint" scenarios, such as security checkpoints. Outdoor tracking remains the most difficult due to unpredictable lighting and crowd density. Furthermore, many high-performing trackers still require manual initialization. Future work must focus on tighter integration between automated detection and long-term tracking to achieve fully autonomous systems.

Summary updated

BibTeX
@inproceedings{whiten2012current,
  author = {Chris Whiten and Wassim Bouachir and Guillaume-Alexandre Bilodeau and Robert Laganière},
  title = {{The current state of face tracking for video surveillance}},
  booktitle = {R-libre},
  year = {2012},
}
Figure