Abstract
Heatmap regression (HR) has become one of the mainstream approaches for face alignment and has obtained promising results under constrained environments. However, when a face image suffers from large pose variations, heavy occlusions and complicated illuminations, the performances of HR methods degrade greatly due to the low resolutions of the generated landmark heatmaps and the exclusion of important high-order information that can be used to learn more discriminative features. To address the alignment problem for faces with extremely large poses and heavy occlusions, this paper proposes a heatmap subpixel regression (HSR) method and a multi-order cross geometry-aware (MCG) model, which are seamlessly integrated into a novel multi-order high-precision hourglass network (MHHN). The HSR method is proposed to achieve high-precision landmark detection by a well-designed subpixel detection loss (SDL) and subpixel detection technology (SDT). At the same time, the MCG model is able to use the proposed multi-order cross information to learn more discriminative representations for enhancing facial geometric constraints and context information. To the best of our knowledge, this is the first study to explore heatmap subpixel regression for robust and high-precision face alignment. The experimental results from challenging benchmark datasets demonstrate that our approach outperforms state-of-the-art methods in the literature.
Introduction
Face alignment—the task of locating predefined facial landmarks like the corners of the eyes, nose tip, and mouth—is a foundational step for downstream tasks like facial recognition and 3D reconstruction. Currently, Heatmap Regression (HR) is the dominant approach. However, HR methods face two major bottlenecks:
- Resolution Limits: HR models typically output low-resolution heatmaps (e.g., 64x64). Simply picking the pixel with the highest value only provides pixel-level accuracy. When mapped back to the original high-resolution image, this introduces significant quantization errors.
- Loss of Geometric Context: Standard HR models often struggle with heavy occlusions or extreme poses because they fail to capture the higher-order spatial correlations between different facial features (e.g., inferring a hidden left eye based on the right eye and nose). Furthermore, standard Mean Squared Error (MSE) loss tends to produce blurry, imprecise heatmaps.
Methodology
To solve these issues, the authors propose a unified architecture, the Multi-order High-precision Hourglass Network (MHHN), built on a stacked hourglass network backbone and enhanced by two novel modules:
Heatmap Subpixel Regression (HSR)
Instead of using standard MSE loss, HSR introduces a Subpixel Detection Loss (SDL) based on Jensen-Shannon divergence. This forces the network to focus on the foreground peak of the heatmap, generating sharper, more accurate distributions. To extract the exact coordinate, HSR uses Subpixel Detection Technology (SDT), which fits a 2D Gaussian distribution over the heatmap's peak. The center of this continuous Gaussian curve becomes the predicted landmark, allowing for subpixel accuracy without the computational cost of generating ultra-high-resolution heatmaps.
Multi-order Cross Geometry-aware (MCG) Model
To handle occlusions and large poses, the MCG model fuses cross-layer information (features from different network depths) with cross-order information. It calculates the outer product of feature maps to model 2nd- and 3rd-order spatial correlations. This acts as a powerful structural attention mechanism, enforcing global geometric constraints even when local features are hidden.
Experimental Results
The authors evaluated their model on four widely used and highly challenging face alignment benchmarks: 300W (Challenging subset), COFW (heavy occlusions), AFLW (large poses), and WFLW (complex environmental factors).
The proposed MHHN consistently outperformed state-of-the-art methods (like LAB, ODN, and SAN) across the board:
- 300W: Achieved a Normalized Mean Error (NME) of 6.01% on the Challenging subset.
- COFW: Demonstrated extreme robustness to occlusion, dropping the failure rate (images with >10% error) to just 1.78%, compared to previous bests around 2.76%.
- AFLW & WFLW: Achieved 1.38% NME on AFLW (full) and 4.77% NME on WFLW, proving superior handling of large poses.
Ablation studies confirmed that HSR can be used to reduce computational cost. By shrinking the input/output resolution by half, the NME on the 300W Challenging set only rose slightly (from 6.01% to 6.07%), which still beats existing SOTA models.
Conclusion
The MHHN architecture effectively addresses the limitations of standard heatmap regression through subpixel precision and higher-order geometric modeling. While the MCG model's use of outer products introduces additional computational overhead, the authors mitigate this by showing the HSR module allows for smaller input/output sizes without severe performance drops. The results demonstrate that capturing structural "rules" of the face through higher-order features is key to handling extreme occlusions and poses. Future work suggests that the HSR framework is highly generalized and could be applied to any heatmap-regression task in computer vision, such as human pose estimation or object segmentation.
Summary updated