Phased Groupwise Face Alignment

Gang Zhang, Han Qin, Yuding Ke, Jiansheng Chen, Yanmin GongIEEE Access 2020 · OpenReview

Abstract

A face does not only have rigid variations but also non-rigid distortions, which has influenced the performance of groupwise face alignment. A novel method for groupwise face alignment which considers both rigid variations of a face and non-rigid distortions was presented in the paper. The process for groupwise face alignment was divided into two stages, i.e. affine transformations and non-rigid distortions. At the stage of the affine transformations, the key points of a face were categorized into five groups and the affine transformations were used for each group of the key points. At the stage of the non-rigid distortions, a novel method was used for all of the key points in a face. Two stages were independent of each other, and moreover, iterations were made in each stage. Besides, the results from the stage of the affine transformations were used as the input of the non-rigid distortion stage. Experiments show that the method for groupwise face alignment in the paper is better than that only considering global affine variations, and is also better than that considering global affine variations and local non-rigid distortions.

Introduction

Face alignment—the process of identifying the geometric structure of a face through "key points" or landmarks—is a cornerstone of facial analysis. Groupwise alignment takes this a step further by considering multiple images simultaneously, using collective information to improve the alignment of each individual face.

The core challenge lies in the dual nature of facial variation. Faces undergo rigid variations (global changes like rotation, scaling, and translation) and non-rigid distortions (local changes caused by expressions or individual anatomy). Most existing methods either treat the face as a single rigid object or struggle to balance global and local constraints. If constraints are too strict, the optimization process can fail; if they are too loose, the landmarks might "shrink" toward the center of the face during iterative updates, a common failure mode in groupwise optimization.

Methodology: The Phased Approach

The authors propose a "phased" approach that treats rigid and non-rigid variations as two independent, sequential stages:

Stage 1: Grouped Affine Transformations

Instead of applying a single affine transformation to the entire face, the authors categorize key points into five functional groups: the outer contour (O_C), left eye (L_E), right eye (R_E), nose (N), and mouth (M).

Landmark grouping into functional units: (a) outer contour, (b) left eye, (c) right eye, (d) nose, and (e) mouth.

Affine transformations are calculated and applied to each group independently. This prevents the rigid constraints of one part of the face (like the jawline) from negatively impacting the alignment of another (like the eyes).

Stage 2: Non-rigid Distortions and Iterative Updates

The results from the first stage serve as the input for the second. Here, a novel method is applied to all key points simultaneously to account for fine-grained, non-linear distortions.

The iterative joint face alignment process, illustrating the interaction between the reference face and individual face updates.

The entire process is iterative. A "reference face" is dynamically updated at each step to act as a constraint. To solve the "shrinking" problem, the authors found that setting a fixed, low number of iterations (specifically five) provided a stable termination condition that prevented the landmarks from collapsing toward the centroid.

Experimental Results

The researchers evaluated their method using the Helen face database, a standard benchmark in the field. They specifically selected front-facing or near-front-facing images to ensure all landmarks were visible. The authors compared their "Phased" method against two baselines: one considering only global affine variations and another considering global variations plus local non-rigid distortions without grouping.

  • Quantitative Performance: Using a Mean Squared Error (MSE) style measure, the proposed method achieved a score of 88,728.075, significantly lower (better) than the global-only method (90,507.549) and the non-grouped method (90,199.883).
  • Qualitative Performance: Visual results showed that the phased approach was much better at capturing the corners of the eyes and the specific curvature of the mouth, areas where traditional methods often drift.

Conclusion

The key insight of this work is that functional grouping is essential for balancing flexibility and structure. By breaking the face into five groups for the affine stage, the model gains the flexibility to handle local rigidities without losing global structure. While the method shows clear improvements, failures still occur when non-rigid distortions are too complex, particularly in the eyebrows (which are currently grouped with the outer contour). Future work will focus on refining these groups and investigating more sophisticated methods for fitting highly non-linear distortions.

Summary updated

Abstract

A face does not only have rigid variations but also non-rigid distortions, which has influenced the performance of groupwise face alignment. A novel method for groupwise face alignment which considers both rigid variations of a face and non-rigid distortions was presented in the paper. The process for groupwise face alignment was divided into two stages, i.e. affine transformations and non-rigid distortions. At the stage of the affine transformations, the key points of a face were categorized into five groups and the affine transformations were used for each group of the key points. At the stage of the non-rigid distortions, a novel method was used for all of the key points in a face. Two stages were independent of each other, and moreover, iterations were made in each stage. Besides, the results from the stage of the affine transformations were used as the input of the non-rigid distortion stage. Experiments show that the method for groupwise face alignment in the paper is better than that only considering global affine variations, and is also better than that considering global affine variations and local non-rigid distortions. The method in the paper can be used as a novel method for groupwise face alignment.
BibTeX
@inproceedings{zhang2020phased,
  author = {Gang Zhang and Han Qin and Yuding Ke and Jiansheng Chen and Yanmin Gong},
  title = {{Phased Groupwise Face Alignment}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=qTBp77EP0mT},
}
Figure