Real-time 3D Facial Tracking via Cascaded Compositional Learning

Jianwen Lou, Xiaoxu Cai, Junyu Dong, Hui YuIEEE TIP 2020 · DOI

Abstract

This paper introduces GoMBF (Globally-optimized Modular Boosted Ferns), a novel regression framework designed for high-fidelity, real-time 3D facial tracking using standard webcam hardware. 3D facial tracking requires estimating multi-modal parameters, including head pose, facial expressions, and landmark displacements, which vary significantly in scale and geometric impact. GoMBF addresses this by decomposing the problem into modular learning tasks followed by a global optimization step. The resulting GoMBF-Cascade achieves state-of-the-art accuracy while training 65.2% faster than traditional boosted ferns and maintaining a consistent 30fps on standard laptop CPUs.

Introduction: The Challenge of Multi-modal Tracking

3D facial tracking involves a complex set of parameters: head rotation and translation, facial expression blendshapes, and landmark displacements. These parameters are inherently multi-modal; for instance, a minor change in head rotation drastically alters the 2D image, whereas a large change in an expression coefficient might produce only subtle visual shifts.

Standard machine learning regressors, such as boosted ferns, often struggle with this variety, becoming "distracted" by high-dimensionality parameters or those with large numerical magnitudes. Previous attempts to solve this relied on manual weighting or computationally expensive deep learning architectures. GoMBF-Cascade provides an efficient alternative that runs on standard hardware without requiring high-end GPUs.

Overview of the GoMBF-Cascade pipeline for 3D facial motion regression.

Methodology: GoMBF-Cascade

The core innovation is the GoMBF-Cascade framework, which utilizes a "divide and conquer" strategy through two primary phases:

  1. Modular Learning (Divide): Instead of a monolithic model, the system trains separate "modular" boosted ferns for each parameter type (e.g., rotation, expression). This specialization ensures that the unique nuances of each modality are captured without interference from others.
  2. Global Optimization (Conquer): These modules are "deeply composed" using Ridge Regression to simultaneously optimize all fern leaves across all modules against a global loss function. This allows the modules to share complementary information—for example, using expression features to help refine landmark positions.
  3. Cascaded Refinement: Multiple GoMBF stages are stacked in a cascade. Each stage takes the output of the previous one and predicts a residual increment, progressively "snapping" the 3D model onto the face in the video frame.
Architecture of Globally-optimized Modular Boosted Ferns (GoMBF) showing (a) the boosted ferns structure and (b) the modular learning and global optimization stages.

Datasets and Evaluation

The model was trained on a combination of real-world and synthetic data:

  • Real Data: Approximately 6,461 images from 300W-3D, FaceWarehouse, and Multi-PIE.
  • Evaluation: The 300VW (300 Videos in the Wild) dataset was used to benchmark accuracy.
  • Synthetic Generalization: The authors tested generalization using three types of synthetic data, ranging from basic 3D models to high-quality textures with HDR lighting and "CoarseData" (real facial images warped onto new geometries).

Key Results and Insights

  • Efficiency and Speed: GoMBF achieves 30fps on a standard i5 laptop CPU. It reduces training time significantly, from ~3,532 seconds per stage for standard Explicit Shape Regression (ESR) to just 1,228 seconds.
  • Tracking Accuracy: The system consistently outperforms ESR in 2D landmark tracking, particularly for challenging movements like mouth opening and eye closure.
  • Robustness: Qualitative testing demonstrates superior resilience to occlusions (e.g., hands covering the face) compared to previous non-deep learning methods.
  • The Synthetic Data Paradox: Interestingly, the study found that for boosted fern regressors, purely synthetic training generalizes poorly. While adding synthetic data can help in specific cases, it often fails to capture the "messiness" of real-world video as effectively as real data.

Limitations and Future Work

The current model relies on an initial estimate of identity and focal length from the first frame; if this initialization is poor, tracking quality may degrade. Additionally, the synthetic data used lacks internal mouth structures and fine-grained variations. Future work may involve using Generative Adversarial Networks (GANs) to bridge the gap between synthetic and real-world training data.

Summary updated

BibTeX
@article{lou2020real,
  author = {Jianwen Lou and Xiaoxu Cai and Junyu Dong and Hui Yu},
  title = {{Real-time 3D Facial Tracking via Cascaded Compositional Learning}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  doi = {10.1109/tip.2021.3065819},
}
Figure