Towards Arbitrary-View Face Alignment by Recommendation Trees

Shizhan Zhu, Cheng Li, Chen Change Loy, Xiaoou TangarXiv 2015 · arXiv

Face alignment—the task of identifying the geometric structure of a face via landmarks like eyes, nose, and mouth—is a cornerstone of facial analysis. However, most existing methods struggle when faces turn away from the camera. This paper introduces a unified framework that handles everything from frontal to full-profile views (up to $\pm 90^\circ$ yaw) by replacing error-prone, explicit head pose estimation with an Ensemble of Model Recommendation Trees (EMRT).

The Challenge of Profile Views

The authors identify two primary hurdles in arbitrary-view alignment:

  1. Multimodal Appearance-Shape Relations: The visual appearance of a face changes drastically between frontal and profile views, making it difficult for a single mathematical model to map pixels to landmarks across all angles.
  2. Varying Landmark Protocols: In profile views, half the face is hidden due to self-occlusion. This requires different sets of landmarks (e.g., 68 points for frontal vs. 39 for profile), which traditional unified models are not designed to handle.

Previous solutions typically estimated head pose first and then selected a specific model. However, if the pose estimation is slightly inaccurate, the subsequent alignment fails catastrophically.

The Proposed Framework

To overcome these issues, the authors propose a three-step framework centered on a "soft selection" mechanism:

  1. Model Pool Construction: A set of $C$ specialized discriminative regression models is trained. Each model is an expert for a specific cluster of poses or landmark protocols (e.g., left-profile, frontal, etc.).
  2. Recommendation Feature Extraction: Instead of using raw pixels, the system runs the test image through the model pool to extract "shape responses." These are combined with part detector scores (SIFT+SVM) to create a feature vector that is highly discriminative for pose.
  3. Ensemble of Model Recommendation Trees (EMRT): This core innovation produces a rating vector rather than a single classification label. This vector acts as a set of weights to intelligently combine the outputs of the entire model pool.
Overview of the proposed framework: (a) shape responses are extracted from a model pool, (b) features are formed, and (c) recommendation trees produce a rating vector for the final estimate.

Direct Optimization

Unlike standard decision trees that split nodes based on information gain (entropy), EMRT splits nodes to directly minimize landmark estimation error. This ensures that faces with similar geometric shapes are clustered together, and models that are "semantically near" (e.g., models for $+30^\circ$ and $+45^\circ$ yaw) naturally share similar ratings.

Key Results and Insights

The researchers evaluated their method on three major benchmarks: AFLW (Annotated Facial Landmarks in the Wild), MultiPIE, and AFW.

  • AFLW Superiority: On the full AFLW dataset, which includes extreme profile views, the proposed method achieved a mean error of 4.01, significantly outperforming standard methods like SDM (6.98) and LBF (7.47).
  • Soft Selection Advantage: By using a rating vector instead of a single model pick, the system can blend models. This provides a "safety net" if the face falls between two trained pose clusters.
  • Visibility Accuracy: The system achieved 91.18% accuracy in detecting whether a landmark was visible or occluded, a critical feature for downstream tasks like 3D face reconstruction.
Qualitative comparison of face alignment results across different views. "Ours" maintains high accuracy in profile views where baselines like Zhu et al. and SDM often fail.

Conclusion

By moving away from hard pose classification and toward a recommendation-based ensemble, this framework provides a robust solution for arbitrary-view face alignment. While the initial "shape response" extraction adds some computational complexity, the gain in accuracy and the ability to handle extreme poses make it a significant advancement for "in-the-wild" facial analysis.

Summary updated

Abstract

Learning to simultaneously handle face alignment of arbitrary views, e.g. frontal and profile views, appears to be more challenging than we thought. The difficulties lay in i) accommodating the complex appearance-shape relations exhibited in different views, and ii) encompassing the varying landmark point sets due to self-occlusion and different landmark protocols. Most existing studies approach this problem via training multiple viewpoint-specific models, and conduct head pose estimation for model selection. This solution is intuitive but the performance is highly susceptible to inaccurate head pose estimation. In this study, we address this shortcoming through learning an Ensemble of Model Recommendation Trees (EMRT), which is capable of selecting optimal model configuration without prior head pose estimation. The unified framework seamlessly handles different viewpoints and landmark protocols, and it is trained by optimising directly on landmark locations, thus yielding superior results on arbitrary-view face alignment. This is the first study that performs face alignment on the full AFLWdataset with faces of different views including profile view. State-of-the-art performances are also reported on MultiPIE and AFW datasets containing both frontaland profile-view faces.
BibTeX
@misc{zhu2015towards,
  author = {Shizhan Zhu and Cheng Li and Chen Change Loy and Xiaoou Tang},
  title = {{Towards Arbitrary-View Face Alignment by Recommendation Trees}},
  year = {2015},
  eprint = {1511.06627},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1511.06627},
}
Figure