TL;DR
This paper presents a novel method for automatically fitting a 3D deformable face mask to a single 2D image, even under challenging conditions like low resolution and varying illumination. By combining a PCA-based face model with a particle filter and a unique texture mapping technique, the approach implicitly detects facial features and achieves robust alignment, outperforming traditional Active Shape Models.
What Problem Does This Paper Address?
Accurately detecting and aligning facial features is a crucial preprocessing step for many computer vision tasks, including face recognition and 3D face modeling. However, real-world images often present significant challenges: faces can be rotated, appear at low resolutions, and be subjected to diverse lighting conditions. These factors make robust and automatic face alignment a difficult problem. Traditional methods, like Active Shape Models (ASM), often rely on manual initialization or struggle with these variations, leading to inaccurate results or complete failure. This paper aims to address these limitations by proposing a method that can automatically fit a deformable 3D face mask to a single image, handling low resolution and varying illumination without requiring manual intervention or prior knowledge of lighting.
Methods
The core of the proposed method involves two main stages: offline model building and online fitting.
Offline Model Building: * 3D Face Model: The authors utilize the CANDIDE-3 face model, a 3D deformable model with 104 vertices and 184 triangles. To increase flexibility for different image resolutions, this model is subdivided using the Modified Butterfly algorithm, resulting in finer meshes. * PCA-based Face Model (Eigenfaces): Instead of using raw image data, the method fits the deformable mask to a set of training images. For each mask triangle, grey values are extracted, and Principal Component Analysis (PCA) is applied to these vectors to create a set of "Eigenfaces." This captures the principal modes of variation in face shape and texture. * Illumination Invariance: To handle varying lighting, the paper incorporates a technique from Basri and Jacobs [3], which represents an image as a linear combination of harmonic images. This representation is integrated into the particle filter's weighting function.
Online Fitting: * Face Detection and Initialisation: A pre-processing step uses a face detector to identify near-frontal faces. The image is then cropped, and the neutral face mask is roughly initialized to fill this window. A fast grid-search is then performed to estimate an initial depth (z-value) for the mask. * Particle Filter: A particle filter is employed to refine the pose (rotation and translation) and shape parameters of the deformable mask. The particle filter iteratively samples potential states (pose and shape) and weights them based on how well the projected 3D mask, rendered with estimated lighting, reconstructs the input image. The error function used for weighting is the reconstruction error in face space, incorporating the illumination invariance technique. * Texture Mapping: A novel texture mapping technique is used where the grey values of each mask triangle are directly used, avoiding the need for complex warping and interpolation typically associated with texture mapping.
Datasets
The paper evaluates its method on two publicly available datasets:
- IMM Face Database [17]: This dataset contains 240 images of 40 individuals, with 6 images per individual exhibiting variations in pose, expression, and lighting. For training, the first image ('full frontal face, neutral expression, diffuse light') and the fifth image ('full frontal face, neutral expression, spot light') were used.
- Extended Yale Face Database B [12]: This database includes images of 10 persons under 9 different poses and 64 different illumination conditions, with an extension adding 28 more individuals. The images are further divided into subsets based on the angle between the light source and the camera axis.
Key Insights and Ideas
- Implicit Feature Detection: The method cleverly bypasses the explicit step of detecting facial feature points. By fitting a deformable mask, the positions of key facial landmarks are implicitly determined as a byproduct of the mask alignment process.
- Resolution Independence: The use of a subdivided mask allows the face model, trained at a high resolution, to be applied to images of significantly lower resolutions, a capability not inherent in many traditional methods.
- Illumination Robustness through Harmonic Images: Incorporating the harmonic image representation into the particle filter's weighting function is a key innovation for achieving illumination invariance, making the fitting process more robust to varying lighting conditions.
- Combined Pose and Shape Estimation: The particle filter effectively estimates both the 3D pose of the face and the person-specific shape parameters simultaneously, leading to a more accurate and complete 3D alignment.
Key Results
- Performance on IMM Face Database: The proposed method achieved a mean vertex difference of 2.7 pixels on the IMM Face Database when tested on training data, significantly outperforming ASM which achieved 6.2 pixels on average for acceptable fits.
- Generalization: The method demonstrated good generalization capabilities, with only a slight increase in mean vertex difference (4.2 pixels) for a "leave-one-out" experiment on the IMM dataset.
- Performance on Yale Face Database B: The approach showed promising results on the Yale Face Database, with mean vertex differences of 6.2 and 6.5 pixels for different subsets, even though the model was trained solely on the IMM dataset and the Yale dataset presented more challenging lighting and poses.
- Resolution Impact: Results indicated that finer masks and higher input image resolutions lead to more accurate fitting. However, the method still performed reasonably well on lower resolutions.
Limitations and Future Work
- Limitations:
- The paper notes that some shape parameters are estimated more precisely than others. Parameters controlling positions (e.g., 'nose vertical position') are more accurately estimated than those affecting smaller regions (e.g., 'eyebrows vertical position', 'mouth width').
- The method is sensitive to initialization, especially for ASM. While the proposed method includes an initialization step, extreme lighting or unusual poses might still pose challenges.
- The performance degrades with more extreme lighting conditions and lower resolutions.
- Future Work: The authors suggest extending the approach to handle facial expressions and partial occlusions of the face.
Summary updated