Robust 3D Face Alignment with Multi-Path Neural Architecture Search

Zhichao Jiang, Hongsong Wang, Xi Teng, Baopu Li

Abstract

3D face alignment is a very challenging and fundamental problem in computer vision. Existing deep learning-based methods manually design different networks to regress either parameters of a 3D face model or 3D positions of face vertices. However, designing such networks relies on expert knowledge, and these methods often struggle to produce consistent results across various face poses. To address this limitation, we employ Neural Architecture Search (NAS) to automatically discover the optimal architecture for 3D face alignment. We propose a novel Multi-path One-shot Neural Architecture Search (MONAS) framework that leverages multi-scale features and contextual information to enhance face alignment across various poses. The MONAS comprises two key algorithms: Multi-path Networks Unbiased Sampling Based Training and Simulated Annealing based Multi-path One-shot Search. Experimental results on three popular benchmarks demonstrate the superior performance of the MONAS for both sparse alignment and dense alignment.

Introduction

3D face alignment involves estimating the precise 3D shape and pose of a face from a single 2D image. While deep learning has made significant strides, most existing models rely on architectures designed by human experts. These "hand-crafted" networks often struggle to maintain consistency across diverse facial poses, occlusions, and lighting conditions. Because different facial regions and poses require features at different scales, finding the optimal way to fuse these features is a complex challenge for human designers.

To address this, researchers from Baidu and Southeast University developed MONAS (Multi-path One-shot Neural Architecture Search), the first framework to automatically search for the optimal neural network design for 3D face alignment. By moving away from manual design, MONAS achieves state-of-the-art accuracy, particularly in challenging scenarios with extreme head poses.

Methods

The MONAS framework utilizes a "One-shot" approach where a single massive "supernet" contains all possible paths, and individual "child" networks are sampled from it during the search process.

The MONAS pipeline: from input face image to UV position map regression via a multi-path one-shot NAS architecture.

Key components of the framework include:

  • Multi-path Search Space: A grid-based search space that allows connections between any scales (downsampling, upsampling, or peer-to-peer), utilizing specialized HPM (Hierarchical, Parallel, and Multi-scale) blocks.
  • MNUS (Multi-path Networks Unbiased Sampling): A novel training strategy ensuring every possible connection in the supernet is sampled and trained equally, preventing bias toward specific paths.
  • SAMOS (Simulated Annealing based Multi-path One-shot Search): A probabilistic technique used to efficiently navigate the search space and identify the best-performing child network after the supernet is trained.
  • Regression Target: The model regresses a UV position map, which records 3D coordinates in a 2D image space.

Experimental Setup and Datasets

The model was trained on 300W-LP (60,000 synthetic large-pose images) and evaluated on several industry-standard benchmarks:

  • AFLW2000-3D: For sparse alignment (68 landmarks) across small, medium, and large poses.
  • AFLW-LFPA: For sparse alignment with balanced yaw angles.
  • Florence: For dense 3D face alignment evaluation (19k vertices) using accurate 3D scans.

Key Results and Insights

The search algorithm automatically discovered a structure with three distinct phases: a downward-scale fusion stage, a full-scale fusion stage, and an upward-scale fusion stage. This validates the intuition that multi-scale processing is vital for 3D tasks.

Normalised Mean Error across different yaw angles, demonstrating MONAS's superior robustness in large-pose scenarios compared to existing methods.

MONAS significantly outperformed manually designed multi-scale networks like HRNet and Hourglass. On the AFLW2000-3D dataset, MONAS achieved a Mean Normalized Mean Error (NME) of 3.19%, outperforming PRNet (3.62%) and 3DDFA V2 (3.51%). The performance gain was most dramatic in "large pose" scenarios (60° to 90° yaw), where MONAS reduced the error significantly compared to previous state-of-the-art methods. On the Florence dataset, MONAS achieved the lowest NME (3.51) for dense 19k-vertex reconstruction.

Despite the complexity of the search, the process is efficient; the search phase takes only 1.6 GPU hours, and the final model is faster to train than traditional supernet approaches.

Conclusion

MONAS demonstrates that automated architecture search can discover more efficient "wiring" for 3D face alignment than expert human design. While the current search is limited to topology, future work could expand to include kernel sizes or activation functions. Additionally, further exploration into real-time inference latency on edge devices will be crucial for practical applications.

Summary updated

Abstract

3D face alignment is a very challenging and fundamental problem in computer vision. Existing deep learning-based methods manually design different networks to regress either parameters of a 3D face model or 3D positions of face vertices. However, designing such networks relies on expert knowledge, and these methods often struggle to produce consistent results across various face poses. To address this limitation, we employ Neural Architecture Search (NAS) to automatically discover the optimal architecture for 3D face alignment. We propose a novel Multi-path One-shot Neural Architecture Search (MONAS) framework that leverages multi-scale features and contextual information to enhance face alignment across various poses. The MONAS comprises two key algorithms: Multi-path Networks Unbiased Sampling Based Training and Simulated Annealing based Multi-path One-shot Search. Experimental results on three popular benchmarks demonstrate the superior performance of the MONAS for both sparse alignment and dense alignment.
BibTeX
@inproceedings{jiang2024robust,
  author = {Zhichao Jiang and Hongsong Wang and Xi Teng and Baopu Li},
  title = {{Robust 3D Face Alignment with Multi-Path Neural Architecture Search}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10687780},
}
Figure