Joint Alignment of Image Faces

Gang Zhang, Lijia Pan, Jiansheng Chen, Yanmin Gong, Fuyuan LiuIEEE Access 2020 · OpenReview

Abstract

Research in face alignment has advanced significantly through the use of prior information and auxiliary models. However, the inherent lack of information in single face images remains a bottleneck. This paper presents a joint alignment method that considers multiple face images simultaneously to optimize alignment results. The proposed approach iteratively performs rigid variations and non-rigid distortions. A key innovation is the division of facial landmarks into five distinct groups during the rigid variation stage to mitigate the restrictive effects of global shape constraints. Experimental results demonstrate that this local grouping strategy achieves superior precision compared to traditional phased or iterative global alignment methods.

Introduction

Face alignment—the task of identifying precise coordinates for facial features such as the eyes, nose, and mouth—is fundamental to facial recognition, expression analysis, and tracking. While single-image alignment methods have improved with deep learning, they often struggle with noise, occlusion, and lighting variations that obscure critical landmarks.

Joint alignment offers a solution by leveraging shared information across a set of images. However, many existing joint methods treat the face as a single rigid entity during initial optimization. This "global constraint" can be overly restrictive, causing local features to be misaligned as the model attempts to fit the entire face to a mean shape. This paper addresses this "rigidity trap" by introducing a Local Grouping Strategy (LGS) within an iterative framework.

Methodology

The proposed method utilizes an iterative optimization process where rigid and non-rigid transformations are performed sequentially in each loop.

Initial Alignment and Reference Face

The process begins by applying the Supervised Descent Method (SDM) to each face image in the collection to obtain a baseline set of coordinates. A reference "mean face" is established using these initial results.

The Local Grouping Strategy (LGS)

The core innovation lies in how rigid transformations are handled. Instead of applying a single transformation to all 68 facial landmarks, the points are divided into five functional groups: outer contour, left eye, right eye, nose, and mouth.

Flowchart of the proposed joint alignment process, illustrating the iterative optimization of rigid and non-rigid transformations.

Iterative Optimization

As shown in the flowchart above, the alignment proceeds through several stages:

  1. Rigid Stage: RANSAC is employed to find the optimal rigid transformation for each of the five groups independently. This allows local features to align more accurately with the reference face without being pulled out of place by global offsets.
  2. Non-Rigid Stage: The results from the rigid stage serve as input for non-rigid distortion adjustments, which refine the landmark positions further.
  3. Update Cycle: After each iteration, the reference face is updated using General Procrustes Analysis based on the newly aligned coordinates. This cycle repeats until the solution stabilizes.

Experimental Evaluation

The researchers evaluated the LGS method using the Helen face database, selecting images where all 68 landmarks were visible to ensure a rigorous comparison.

Comparative Analysis

The Local Grouping Strategy was compared against two baseline approaches:

  • Phased Global (GGP): Rigid and non-rigid transformations are performed in separate, non-iterative phases using global constraints.
  • Iterative Global (GGS): Rigid and non-rigid transformations are performed iteratively, but the rigid stage uses a single global constraint for the entire face.

Results

The LGS method significantly outperformed both global strategies. In a test involving a set of 5 images, LGS achieved an error score of 4230.95, compared to 5885.82 for GGP and 6507.53 for GGS. Furthermore, the study found that as the number of images in the joint set increased (from 3 to 11), the alignment error generally decreased for all methods, confirming that joint optimization effectively utilizes cross-image information to improve convergence.

Qualitatively, the LGS method proved superior in maintaining the integrity of local features, particularly at the corners of the eyes and the contours of the mouth, where global methods typically exhibit "drift."

Conclusion

By decomposing the rigid alignment stage into localized groups, this research provides a more flexible and accurate framework for joint face alignment. The iterative nature of the process ensures that both rigid and non-rigid variations are optimized in concert. While the method's performance is currently dependent on the quality of the initial SDM alignment, it represents a significant step forward in multi-image facial analysis. Future work will explore enhancing robustness against poor initializations and extreme facial poses.

Summary updated

Abstract

Researches on face alignment have made great progress, which benefits from the use of prior information and auxiliary models. However, that information lacks in a single face image has always affected the further development of these researches. The methods considering multiple face images provide a feasible way to solve the problem undoubtedly. Joint alignment where multiple face images are considered was presented in the paper. Face alignment was used for each face, and joint face alignment was used for optimizing the alignment results of all faces further. During joint alignment, both rigid variations of faces and non-rigid distortions were considered, however, they were regarded as two independent stages. Joint face alignment was a process where optimization was performed iteratively. In each iteration, both rigid variations and non-rigid distortions were performed sequentially, and moreover, the results of rigid variations were used as input of non-rigid distortions. At the stage of rigid variations, the key points of a face were divided into five groups to reduce the effect of global constraints which was imposed by face shape. After several iterations, the optimal solution of joint alignment can be obtained. The experimental results show that the joint alignment can obtain the optimal results than joint alignment using phased global rigid variations and non-rigid distortions and that using iterative global rigid variations and non-rigid distortions, and it can be used as a novel method for joint alignment.
BibTeX
@inproceedings{zhang2020joint,
  author = {Gang Zhang and Lijia Pan and Jiansheng Chen and Yanmin Gong and Fuyuan Liu},
  title = {{Joint Alignment of Image Faces}},
  booktitle = {IEEE Access},
  year = {2020},
  url = {https://openreview.net/forum?id=vb3lxLsHz30},
}
Figure