Abstract
Accurate tracking of facial expressions in 3D sequences is vital for applications in animation, medicine, and human-computer interaction. This paper presents a robust framework for dense 3D facial tracking that addresses the limitations of static fitting methods. By extending the Non-rigid Iterative Closest Point (NICP) algorithm with a Kalman filter-based motion history constraint, we achieve high temporal consistency. Our multi-stage pipeline—combining 3D Morphable Model (3DMM) initialization with local geometric refinement—outperforms existing baselines on the BU4D-FE database, reducing coordinate error and improving surface normal accuracy.
Introduction
Facial tracking has traditionally relied on 2D imagery, which is susceptible to environmental factors like lighting and occlusions. While 3D scans provide geometric depth, they present the challenge of "dense tracking"—establishing point-to-point correspondences for thousands of vertices across a dynamic sequence. Existing 3D methods often fall into two categories: sparse landmark tracking, which lacks detail, or static dense fitting, which ignores temporal flow and leads to jittery results. We propose a framework that bridges this gap by incorporating temporal motion priors into the non-rigid fitting process.
Methodology
Our approach integrates global statistical models with local geometric optimization through a four-stage pipeline.
Joint 2D/3D Landmark Detection
Reliable initialization requires accurate facial landmarks. We utilize a hybrid approach: 3D scans are projected into 2D depth images where standard landmark detectors are applied. These 2D coordinates are then mapped back into 3D space using an inverse Z-buffer technique, providing stable anchors for subsequent fitting stages.
3DMM Initialization and Rigid Alignment
To establish a global shape, a 3D Morphable Model (3DMM) is fitted to the first frame. This provides a statistically plausible starting point. We then apply a rigid Iterative Closest Point (ICP) algorithm with a scale factor to align the template's orientation and size with the target scan.
Non-rigid ICP with Motion History
The core innovation lies in the temporal regularization of the Non-rigid ICP (NICP) algorithm. Standard NICP optimizes a cost function based on vertex distance, stiffness, and landmark alignment. We introduce a fourth term: Motion History. By employing a Kalman filter, we predict the position of each vertex in the current frame based on its trajectory in previous frames. This prevents unnatural mesh warping and ensures smooth transitions, even during rapid facial movements.
Experimental Evaluation
The framework was evaluated using the BU4D-FE (Binghamton University 3D Dynamic Facial Expression Database), featuring 101 subjects performing six prototypical expressions.
Quantitative Results
The proposed method achieved a coordinate error of 8.6 (units of $10^{-3}$), significantly lower than standard NICP (9.9) and 3DMM fitting (11.3). Surface normal error was also minimized to 0.31, indicating superior reconstruction of fine facial details.
Qualitative Improvements
Visual analysis demonstrates that our method maintains surface smoothness and detail in regions of high deformation, such as the mouth and eyelids, where traditional 3DMMs often fail due to their limited PCA basis. The temporal constraint effectively eliminates the "jitter" common in frame-by-frame fitting approaches.
Conclusion and Future Work
By treating facial motion as a predictable physical property, our framework achieves significant improvements in the stability and accuracy of dense 3D tracking. Future research will explore the integration of texture-based optical flow to further refine correspondences and investigate optimizations for real-time performance.
Summary updated