TL;DR
This paper introduces a novel "intrinsic normalization" technique for 3D facial data. By normalizing facial geometry to a template, it removes subject-specific physiognomy variations, leading to more accurate and robust detection of facial action units (AUs) and reducing intra-class variance.
What Problem Does This Paper Address?
Accurately detecting facial action units (AUs) from 3D facial scans is crucial for understanding human expressions, with applications ranging from mental health diagnostics to human-computer interaction. A major challenge in this field is the significant variability introduced by individual facial physiognomy (e.g., face shape, bone structure). Existing methods often struggle to disentangle these inherent subject differences from the actual facial movements (AUs) that convey expression. This paper aims to address this by developing a method that normalizes facial geometry to be independent of individual physiognomy, thereby allowing for more accurate AU detection.
Methods
The core contribution is the proposed "intrinsic normalization" framework. This method leverages a geometric approach, treating facial surfaces as Riemannian manifolds. The key idea is to deform an input neutral face and an expressive face to match a template mesh, effectively transferring the expression onto the template's physiognomy. This is achieved through a "deformation transfer" algorithm, which uses gradient fields to ensure that the expression's geometric changes are applied to the template. Specifically, the process involves:
- Registration: Aligning input neutral and expressive face meshes (S and S') to a template mesh (T) using non-rigid registration (NRR).
- Deformation Transfer: Using the deformation D that transforms S to S' to deform the template mesh T into a normalized expressive face T'. This process aims to make T' have the physiognomy of T and the expression of S'.
- Smoothing: Applying Laplace-Beltrami smoothing to the meshes to mitigate noise introduced during registration and deformation transfer, while preserving geometric integrity.
This process effectively creates a representation of the expression that is independent of the original subject's physiognomy.
Datasets
The primary dataset used for evaluation is the Bosphorus database. This database contains 3D facial scans of 105 subjects, with each subject performing up to 24 AUs, resulting in a total of 4666 facial scans. Each scan includes a depth map, an RGB image, and labeled facial landmarks. The authors also mention that their method requires accurate and automatic landmark detection, referencing work that shows this is achievable for 3D data.
Key Insights and Ideas
- Physiognomy as a "Noise" Factor: The paper frames individual facial structure (physiognomy) as a source of variability that hinders AU detection, akin to noise in other data normalization tasks.
- Intrinsic Equivalence of Expressions: The work builds on the idea that expressions of a single subject are "intrinsically equivalent" on the facial manifold, meaning their geometric properties are similar despite surface stretching.
- Deformation Transfer for Expression Normalization: The novel application of deformation transfer to "copy" expression deformations from a subject's face onto a template face is a key methodological insight.
- Subject-Independent Representation: The ultimate goal is to create a representation of facial expressions that is agnostic to the subject's unique physiognomy, enabling more direct comparison and classification of AUs.
Key Results
The paper demonstrates significant improvements in AU detection performance by using their intrinsic normalization technique as a preprocessing step.
- Classification Improvements: When applied to the CS-3DLBP + RBG feature set with Random Forests, the normalized data achieved better ROC AUC error rates across many AUs compared to non-normalized data. For instance, AU4 showed a 65.0% improvement, AU11 a 36.1% improvement, and AU28 a 54.2% improvement.
- Variance Reduction: The method successfully reduced the variance of Gaussian curvature measurements at specific landmarks for faces performing the same AU. For example, for AU9 at marker ID 11, the variance decreased from 2.02e5 to 5.50e3, a substantial reduction. This lower variance is directly linked to improved feature consistency and classification performance.
Limitations and Future Work
- Suboptimal Registration: The authors acknowledge that for some action units, particularly those involving the lips (AU16, AU23) and eyes (AU43), classification performance drops after normalization. They suspect this is due to suboptimal non-rigid registration in these complex areas.
- Manual Neutral Face Capture: While the process of capturing a neutral face is described as trivial, it is still a manual step performed once per subject. Future work could explore automating this.
- Real-time Applications: The paper notes that implementations of NRR and deformation transfer suggest that real-time applications are feasible, hinting at future directions for optimizing the current framework for speed.
- Generalizability: The authors suggest that the framework's application can be generalized beyond facial action unit detection to other action recognition tasks on surfaces with manifold structure, including water-tight surfaces.
Summary updated