Robust Face Alignment by Multi-Order High-Precision Hourglass Network

Jun Wan, Zhihui Lai, Jun Liu, Jie Zhou, Can Gao

TL;DR

This paper introduces the Multi-order High-precision Hourglass Network (MHHN) to tackle face alignment in highly challenging conditions, such as heavy occlusions and extreme poses. By combining a novel Heatmap Subpixel Regression (HSR) technique with a Multi-order Cross Geometry-aware (MCG) model, the authors achieve subpixel landmark accuracy and better capture facial geometric constraints, setting new state-of-the-art results on multiple benchmark datasets.

What Problem Does This Paper Address?

Face alignment—the task of locating predefined facial landmarks like the corners of the eyes, nose tip, and mouth—is a foundational step for downstream tasks like facial recognition and 3D reconstruction. Currently, Heatmap Regression (HR) is the dominant approach. However, HR methods face two major bottlenecks:

  1. Resolution Limits: HR models output low-resolution heatmaps (e.g., 64x64). Simply picking the pixel with the highest value only provides pixel-level accuracy. When mapped back to the original high-resolution image, this introduces significant quantization errors.
  2. Loss of Geometric Context: Standard HR models often struggle with heavy occlusions or extreme poses because they fail to capture the higher-order spatial correlations between different facial features (e.g., inferring a hidden left eye based on the right eye and nose). Furthermore, standard Mean Squared Error (MSE) loss tends to produce blurry, imprecise heatmaps.

Methods

To solve these issues, the authors propose a unified architecture (MHHN) built on a stacked hourglass network backbone, enhanced by two novel modules:

  • Heatmap Subpixel Regression (HSR): Instead of using standard MSE loss, HSR introduces a Subpixel Detection Loss (SDL) based on Jensen-Shannon divergence. This forces the network to focus on the foreground peak of the heatmap, generating sharper, more accurate distributions. To extract the exact coordinate, HSR uses Subpixel Detection Technology (SDT), which fits a 2D Gaussian distribution over the heatmap's peak. The center of this continuous Gaussian curve becomes the predicted landmark, allowing for subpixel accuracy.
  • Multi-order Cross Geometry-aware (MCG) Model: To handle occlusions and large poses, the network needs to understand facial geometry. The MCG model achieves this by fusing cross-layer information (features from different network depths) with cross-order information. It calculates the outer product of feature maps to model 2nd- and 3rd-order spatial correlations. This acts as a powerful structural attention mechanism, enforcing global geometric constraints even when local features are hidden.

Datasets

The authors evaluated their model on four widely used and highly challenging face alignment benchmarks:

  • 300W (68 landmarks): A standard benchmark; the authors specifically focused on the "Challenging" subset.
  • COFW (68 landmarks): A dataset specifically designed to test robustness against heavy occlusions.
  • AFLW (19 landmarks): Features large pose variations and diverse expressions.
  • WFLW (98 landmarks): A highly complex dataset with rich attribute annotations (blur, makeup, illumination, extreme poses).

Key Insights and Ideas

  • Subpixel accuracy doesn't require massive heatmaps: By fitting a 2D Gaussian to the heatmap peak (SDT), the model bypasses the physical pixel grid. This means you can achieve highly precise coordinates without the massive computational cost of generating ultra-high-resolution heatmaps.
  • Jensen-Shannon Divergence beats MSE for heatmaps: MSE treats all pixels equally, which dilutes the network's focus and leads to blurry heatmaps. JS divergence accurately measures the difference between the predicted and ground-truth distributions, forcing the network to generate sharp, convex peaks.
  • Higher-order features act as geometric anchors: By using outer products to capture 2nd- and 3rd-order feature correlations, the network learns the structural "rules" of a face. If half a face is covered by a scarf, the network relies on this learned global geometry to accurately guess the occluded landmarks.

Key Results

The proposed MHHN consistently outperformed state-of-the-art methods (like LAB, ODN, and SAN) across the board:

  • 300W: Achieved a Normalized Mean Error (NME) of 3.18% on the Common subset and 6.01% on the Challenging subset.
  • COFW: Demonstrated extreme robustness to occlusion, dropping the failure rate (images with >10% error) to just 1.78%, compared to previous bests around 2.76%.
  • AFLW & WFLW: Achieved 1.38% NME on AFLW (full) and 4.77% NME on WFLW, proving superior handling of large poses and complex environmental factors.
  • Ablation: The authors proved that HSR can actually be used to reduce computational cost. By shrinking the input/output resolution by half, the NME on the 300W Challenging set only rose slightly (from 6.01% to 6.07%), which still beats existing SOTA models.

Limitations and Future Work

  • Limitations: The MCG model's use of outer products to calculate higher-order features introduces additional computational overhead. While the authors mitigate this by showing the HSR module allows for smaller input/output sizes without severe performance drops, the mathematical operations are inherently more expensive than standard convolutions.
  • Future Work: The authors point out that the Heatmap Subpixel Regression (HSR) framework is highly generalized. It is not limited to face alignment and can theoretically be applied to any heatmap-regression task in computer vision, such as human pose estimation, object segmentation, or foreground-background segmentation.

Summary updated

Abstract

Heatmap regression (HR) has become one of the mainstream approaches for face alignment and has obtained promising results under constrained environments. However, when a face image suffers from large pose variations, heavy occlusions and complicated illuminations, the performances of HR methods degrade greatly due to the low resolutions of the generated landmark heatmaps and the exclusion of important high-order information that can be used to learn more discriminative features. To address the alignment problem for faces with extremely large poses and heavy occlusions, this paper proposes a heatmap subpixel regression (HSR) method and a multi-order cross geometry-aware (MCG) model, which are seamlessly integrated into a novel multi-order high-precision hourglass network (MHHN). The HSR method is proposed to achieve high-precision landmark detection by a well-designed subpixel detection loss (SDL) and subpixel detection technology (SDT). At the same time, the MCG model is able to use the proposed multi-order cross information to learn more discriminative representations for enhancing facial geometric constraints and context information. To the best of our knowledge, this is the first study to explore heatmap subpixel regression for robust and high-precision face alignment. The experimental results from challenging benchmark datasets demonstrate that our approach outperforms state-of-the-art methods in the literature.
BibTeX
@article{wan2021robust,
  author = {Jun Wan and Zhihui Lai and Jun Liu and Jie Zhou and Can Gao},
  title = {{Robust Face Alignment by Multi-Order High-Precision Hourglass Network}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=SjENirUCAKv},
}

Loading related content...

Figure