Capturing Subtle Facial Motions in 3D Face Tracking

Zhen Wen, Thomas S. HuangICCV 2003 · DOI

Abstract

Accurately tracking facial expressions requires capturing both coarse geometric movements and subtle surface deformations like wrinkles. This paper introduces a hybrid 3D face tracking method that utilizes a "ratio-image" feature to isolate expression-induced changes from individual skin tone (albedo). Combined with an online Expectation-Maximization (EM) adaptation algorithm, the system can learn new individuals and lighting environments in real-time. Evaluation on the Cohn-Kanade database shows marked improvements in recognizing complex expressions like anger and disgust.

Introduction

Traditional 3D face tracking typically relies on either geometric features (e.g., tracking the corners of the mouth) or appearance-based methods (tracking pixel intensities). While geometric methods are robust to lighting changes, they often fail to capture subtle motions like the furrowing of a brow or the deepening of a nasolabial fold. Conversely, appearance-based methods can capture these details but often require extensive training data to generalize across different people and environments.

Overview of the tracking process: (a) input frame, (b) geometric mesh tracking, and (c) extraction of subtle facial details.

The proposed framework addresses this by decomposing facial motion into low-frequency geometric deformations and high-frequency subtle details. This allows for a system that is both robust and highly descriptive, capturing the fine-grained details necessary for high-fidelity expression recognition.

Methodology

The core of the approach is the Ratio-Image Feature. By assuming a Lambertian (matte) surface, the ratio of the intensity of a deformed face to a neutral face cancels out the person's specific skin reflectance (albedo). This results in a feature that represents only the change caused by facial motion, making it inherently portable across different ethnicities and skin types.

Feature Extraction and Adaptation

Rather than processing the entire face, the system focuses on 11 key regions where wrinkles and creases are most prominent. Gabor wavelets are applied to these regions to extract multi-scale, multi-orientation texture information.

The 11 facial regions identified for localized high-frequency feature extraction.

To handle environmental variability, the system employs an Online EM Adaptation algorithm. As tracking progresses, the model updates its expression means. To prevent "model drift"—where the tracker loses track of valid facial configurations—updates are constrained to a valid subspace defined by Principal Component Analysis (PCA). This ensures the system learns actual facial expressions rather than absorbing sensor noise.

Experimental Results

The system was evaluated using the CMU Cohn-Kanade Expression Database (2,981 frames, 47 subjects) and custom video sequences featuring extreme rotations and lighting.

Performance Metrics

The inclusion of the ratio-image feature significantly boosted performance in facial expression recognition tasks, particularly for easily confused expressions:

  • Person-Dependent Scenarios: Accuracy for "Anger" rose from 74.8% (geometric only) to 92.7% with the proposed method.
  • Person-Independent Scenarios: The ratio-image feature outperformed standard texture features (37.0% vs 24.7% for Anger).
  • Adaptation Benefits: Online EM adaptation with PCA constraints improved person-independent recognition rates from 70.7% to 77.9%.

Conclusion

By separating albedo from expression-induced texture changes, this method provides a robust way to track subtle facial motions across diverse populations. While the current model assumes a Lambertian surface and lacks temporal dynamics, the significant gains in recognition accuracy demonstrate the power of the ratio-image and online adaptation framework. Future work will focus on modeling non-Lambertian effects and incorporating temporal models like Hidden Markov Models to further smooth tracking.

Summary updated

BibTeX
@inproceedings{wen2003capturing,
  author = {Zhen Wen and Thomas S. Huang},
  title = {{Capturing Subtle Facial Motions in 3D Face Tracking}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2003},
  doi = {10.1109/iccv.2003.1238646},
}
Figure