Online learning of robust facial feature trackers

Tim Sheerman-Chase, Eng-Jon Ong, Richard BowdenWorkshops @ ICCV 2009 · OpenReview

Abstract

This paper presents a head pose and facial feature estimation technique that works over a wide range of pose variations without a priori knowledge of the appearance of the face. Using simple LK trackers, head pose is estimated by Levenberg-Marquardt (LM) pose estimation using the feature tracking as constraints. Factored sampling and RANSAC are employed to both provide a robust pose estimate and identify tracker drift by constraining outliers in the estimation process. The system provides both a head pose estimate and the position of facial features and is capable of tracking over a wide range of head poses.

Introduction

Robustly tracking facial features—such as the corners of the eyes or mouth—is a fundamental challenge in computer vision, underpinning applications from emotion recognition to driver monitoring. The difficulty arises from the dynamic nature of faces: they rotate in 3D space, change expressions, and experience frequent occlusions.

Traditional solutions generally fall into two categories: model-based approaches, which require extensive pre-labeled training data to build generic face models (e.g., Active Appearance Models), and online learning methods, which learn on-the-fly but often suffer from "drift"—the accumulation of small errors that eventually causes the tracker to fail. This research bridges these approaches by introducing a system that requires zero pre-training, instead utilizing the geometric constraints of a 3D head model to stabilize online learning in real-time.

Methodology

The system operates in a continuous five-stage loop to maintain tracking accuracy across varying poses:

  1. Initialization: The process begins with a frontal view where 19 key landmarks are identified. A generic 3D face mesh is projected onto these points to initialize a total of 51 trackers.
  2. Multi-Template Tracking: To handle rotation, the system maintains a "gallery" of templates for each feature point. Lucas-Kanade (LK) tracking is used to locate these features in subsequent frames.
  3. Weighted RANSAC and LM Pose Estimation: The "secret sauce" of the system is the use of the Levenberg-Marquardt (LM) algorithm to fit the 3D mesh to tracked points. To manage "noisy" trackers that have drifted, a modified Weighted Random Sample Consensus (RANSAC) is employed. This version weights points based on their agreement with the 3D model, prioritizing high-confidence trackers for pose estimation.
  4. Tracker Correction: Once a robust pose is determined, the system "pulls" any drifting trackers back to their geometrically correct positions relative to the 3D mesh.
  5. Online Template Update: When the subject reaches a novel head angle, the system captures a new image patch and adds it to the feature's template library, allowing for pose-specific recognition.
Facial feature tracking across different subjects and poses using the proposed system.

Experimental Evaluation

The system was evaluated using the Boston University (BU) head tracking dataset, which contains 45 video sequences of 5 different subjects. This dataset is particularly rigorous as it includes ground truth head pose data recorded via electromagnetic sensors with 0.5-degree accuracy.

Key Results

The system demonstrated performance comparable to state-of-the-art methods that require significantly more training:

  • Pose Accuracy: Average rotation errors were low: Pitch (3.9°), Roll (3.1°), and Yaw (4.2°).
  • Tracking Precision: The average feature tracking error was 3.7 pixels. Notably, manual human annotation of this dataset has a noise level of approximately 2.7 pixels, indicating the algorithm is approaching human-level consistency.
  • Robustness: The system successfully maintained tracking through wide ranges of motion where traditional 2D models typically fail.

Discussion and Future Work

The core insight of this work is that while individual appearance is unknown, the underlying structure of the human face is predictable. By enforcing geometric constraints via a 3D mesh, the authors effectively prevent the drift that typically plagues online learning. Furthermore, the use of pose-specific template libraries allows for high precision during extreme head turns.

However, the reliance on a rigid 3D mesh presents a limitation. The "one-size-fits-all" generic head shape can struggle with subjects whose facial structures deviate significantly from the mean. Additionally, the current model does not account for non-rigid deformations caused by expressions. Future work will explore "variable shape models" that can adapt the 3D mesh to the specific proportions and expressions of the individual subject.

Summary updated

Abstract

This paper presents a head pose and facial feature estimation technique that works over a wide range of pose variations without a priori knowledge of the appearance of the face. Using simple LK trackers, head pose is estimated by Levenberg-Marquardt (LM) pose estimation using the feature tracking as constraints. Factored sampling and RANSAC are employed to both provide a robust pose estimate and identify tracker drift by constraining outliers in the estimation process. The system provides both a head pose estimate and the position of facial features and is capable of tracking over a wide range of head poses.
BibTeX
@inproceedings{sheermanchase2009online,
  author = {Tim Sheerman-Chase and Eng-Jon Ong and Richard Bowden},
  title = {{Online learning of robust facial feature trackers}},
  booktitle = {Workshops @ ICCV},
  year = {2009},
  url = {https://openreview.net/forum?id=Pn5CZPZAkl},
}
Figure