<title>FaceTrack: tracking and summarizing faces from compressed video</title>

Hualu Wang, Harold S. Stone, Shih‐Fu ChangProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 1999 · DOI

TL;DR

This paper introduces FaceTrack, a system designed to automatically detect, track, and group faces within compressed video (MPEG). It leverages a Kalman filter framework with multiple hypothesis tracking to handle challenges like imperfect face detection and multiple moving faces, demonstrating improved accuracy and robustness for video summarization.

What Problem Does This Paper Address?

The explosion of digital video content, particularly in formats like MPEG, presents a significant challenge for users trying to find specific information. Video data is lengthy and tedious to download and navigate. Human faces are crucial visual cues that help understand video content, making face-based searching highly desirable. However, existing face detection and tracking methods often assume uncompressed video and perfect detection, which is unrealistic for real-world applications. This paper tackles the problem of efficiently tracking faces directly from compressed video, addressing issues like false positives, missed detections, and the ambiguity of multiple faces interacting within a scene, ultimately aiming to enable effective video summarization.

Methods

FaceTrack employs a multi-stage approach:

  • Compressed Domain Processing: The system primarily operates in the compressed domain to minimize computational overhead, only decoding frames when necessary for detailed analysis.
  • Scene Change Detection: Initial segmentation of the video into shots is performed using compressed-domain scene change detection algorithms.
  • Face Detection: A specialized face detection algorithm, previously developed by the authors, is used. This algorithm works on a coarse resolution (e.g., 16x16 macroblocks) and utilizes chrominance, shape, and energy distribution constraints in the DCT domain.
  • Face Tracking Framework: The core of the system is a Kalman filter-based framework.
  • Kalman Filter: This is used to predict and update the kinematic state (position, velocity, acceleration) of detected faces. The state vector represents the face's movement, and the observation vector is the detected face's position.
  • Motion Models: The paper compares four motion models for the Kalman filter: constant-velocity (CV), constant-acceleration (CA), correlated-acceleration (AA - proposed by the authors), and variable-dimension filter (VDF). The AA model is designed to better capture the observed dynamics of faces in commercial videos, which often exhibit a tendency to "stabilize."
    • Multiple Hypothesis Tracking (MHT): To resolve ambiguities arising from imperfect face detection (false alarms, misses) and multiple faces interacting, an MHT approach is used. This involves generating multiple hypotheses for track continuation and selecting the most likely ones based on accumulated Mahalanobis distances.
  • Face Summarization: After tracking within shots, faces are grouped across shots using visual features from representative frames and associated body/background regions. This stage is described as being in its early stages.

Datasets

The paper uses commercial videos for evaluation, specifically mentioning:

  • CNN news video: Used for the "Marcia" example.
  • VideoQ database: Used for the "Clinic" and "Couple" examples.

The authors manually identify ground truth face trajectories for evaluation. No new benchmarks are introduced; rather, existing commercial video content is utilized.

Key Insights and Ideas

  • Compressed Domain Tracking is Feasible and Effective: The paper demonstrates that robust face tracking can be achieved directly from compressed video, significantly reducing computational costs compared to pixel-domain methods.
  • Motion Model Choice Matters: For commercial videos with relatively smooth motion and moderate sampling rates, the simple constant-velocity (CV) model or the proposed correlated-acceleration (AA) model are found to be most effective. The constant-acceleration (CA) model, while common in other tracking applications, performs poorly for faces.
  • Multiple Hypothesis Tracking is Crucial for Robustness: Imperfect face detection is a reality. The MHT approach is essential for handling false alarms, missed detections, and the complex interactions of multiple faces, allowing the system to maintain tracks even through temporary occlusions or detection failures.
  • Faces Tend to "Stabilize": The proposed correlated-acceleration (AA) model is motivated by the observation that faces in commercial videos often exhibit a tendency to maintain a relatively smooth motion and "stabilize," rather than constantly accelerating. This model captures this behavior better than simpler models.

Key Results

  • The FaceTrack system effectively reduces face detection errors and resolves tracking ambiguities.
  • Motion Model Performance: For Type A videos (smooth, long movement), the CV and AA models consistently showed the lowest tracking errors (mean distance and max distance) across different noise levels, outperforming CA and VDF. For example, at $\sigma=2.0$, CV and AA had a mean error of 1.42 and 1.43 respectively, compared to 1.50 for CA and 1.46 for VDF.
  • Qualitative Examples: The paper presents detailed examples ("Marcia," "Clinic," "Couple") illustrating how the system handles misses, false alarms, and occlusions, showing the generated face tracks and their recovery. For instance, in the "Marcia" video, despite 2 misses and 6 false alarms, the algorithm successfully recovers the single face track.

Limitations and Future Work

  • Face Summarization is Early Stage: The paper acknowledges that face grouping and summarization across shots is an area of ongoing research. Differentiating faces solely from compressed domain features is challenging, and pixel-domain decoding is often required for robust grouping.
  • Domain Knowledge is Important: For complex scenarios, especially for linking fragmented tracks or disambiguating similar-looking individuals, domain knowledge about the video content (e.g., knowing the number of people, or that clothes don't change) can significantly improve results.
  • Future Research Directions:
    • Optimizing the face tracking algorithm for more adverse situations.
  • Developing more sophisticated face summarization techniques.
  • Integrating face summarization with other video analysis techniques like video scene classification.

Summary updated

Abstract

In this paper, we present FaceTrack, a system that detects, tracks, and groups faces from compressed video data. We introduce the face tracking framework based on the Kalman filter and multiple hypothesis techniques. We compare and discuss the effects of various motion models on tracking performance. Specifically, we investigate constant-velocity, constant-acceleration, correlated-acceleration, and variable-dimension-filter models. We find that constant- velocity and correlated-acceleration models work more effectively for commercial videos sampled at high frame rates. We also develop novel approaches based on multiple hypothesis techniques to resolving ambiguity issues. Simulation results show the effectiveness of the proposed algorithms on tracking faces in real applications.
BibTeX
@inproceedings{wang1999lt,
  author = {Hualu Wang and Harold S. Stone and Shih‐Fu Chang},
  title = {{\<title\>FaceTrack: tracking and summarizing faces from compressed video\</title\>}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {1999},
  doi = {10.1117/12.360426},
}
Figure