Joint Voxel and Coordinate Regression for Accurate 3D Facial Landmark Localization

Hongwen Zhang, Qi Li, Zhenan Sun

Locating facial landmarks in 3D space—such as the corners of the eyes, the tip of the nose, or the jawline—is a fundamental task in computer vision with applications ranging from facial recognition to augmented reality. However, 3D localization is significantly more challenging than its 2D counterpart due to perspective ambiguity and frequent self-occlusions. This paper introduces Joint Voxel and Coordinate Regression (JVCR), an end-to-end framework designed to overcome these hurdles by learning 3D facial structure directly from a single image.

The Challenge: The Curse of Dimensionality

Most existing methods for 3D landmarking follow a "two-step" strategy: first detecting 2D landmarks and then estimating depth or fitting a 3D morphable model. This approach is inherently suboptimal because errors in the initial 2D detection propagate and amplify in the 3D estimation phase.

While heatmap-based regression has seen great success in 2D, extending it to 3D typically requires a separate volumetric heatmap for every landmark. For a standard set of 68 landmarks, the memory and computational requirements become prohibitive—a problem known as the "curse of dimensionality." JVCR addresses this by proposing a more efficient, unified data structure.

Methodology: A Two-Stage Pipeline

The JVCR framework consists of two main components linked in an end-to-end pipeline, utilizing a novel Compact Volumetric Representation.

The JVCR architecture: A coarse-to-fine voxel regression stage followed by a coordinate regression subnetwork.

1. Compact Volumetric Representation

Instead of creating one volume per landmark, the authors propose a single $w \times h \times d$ volume. Each voxel stores the maximum likelihood of any landmark being present at that specific 3D coordinate. This keeps the data structure's size fixed, regardless of whether the model is tracking 5 or 500 landmarks, effectively decoupling model complexity from landmark density.

2. Voxel and Coordinate Regression

  • Voxel Regression Subnetwork: The model uses a Stacked Hourglass Network to estimate the volumetric representation from an input image. It employs a coarse-to-fine strategy, progressively increasing the resolution of the depth ($z$) dimension to handle the inherent difficulty of depth estimation.
  • Coordinate Regression Subnetwork: To extract precise $(x, y, z)$ coordinates, the estimated volume is passed through a 3D Convolutional Neural Network (CNN). Unlike simple "peak picking," the 3D CNN allows the network to learn global structural constraints, enabling the model to infer the position of occluded landmarks based on the visible parts of the face.

Key Results and Performance

The authors validated JVCR using standard benchmarks including 3DFAW, 300W-LP, and AFLW2000-3D. The method achieved state-of-the-art performance, particularly in handling large pose variations.

Comparison of Ground Truth Error (GTE) between JVCR and the FAN+Depth baseline on the 3DFAW dataset.

On the 3DFAW test set, JVCR achieved a Ground Truth Error (GTE) of 4.35%, outperforming previous top-ranked methods. Furthermore, the model demonstrated high efficiency, processing images in approximately 50ms on a TITAN Xp GPU, making it suitable for near-real-time applications.

Conclusion

By breaking the linear scaling of 3D heatmaps and incorporating structural awareness through 3D convolutions, JVCR provides a robust solution for 3D facial alignment. While the compact representation requires a secondary coordinate regression step to "re-identify" specific landmarks, the framework's ability to handle extreme poses and occlusions marks a significant step forward. Future work may extend this joint regression approach to other 3D tasks, such as general object alignment and human pose estimation in the wild.

Summary updated

BibTeX
@inproceedings{zhang2018joint,
  author = {Hongwen Zhang and Qi Li and Zhenan Sun},
  title = {{Joint Voxel and Coordinate Regression for Accurate 3D Facial Landmark Localization}},
  booktitle = {ICPR},
  year = {2018},
  doi = {10.1109/icpr.2018.8546220},
}
Figure