3D face imaging and reconstruction technology: a review

Fei Liu, Kunbo Zhang, Qing Yang, Shubo Zhou, Yunlong Wang, Zhenan SunJournal of Image and Graphics 2024 · DOI

Abstract

得益于新型三维视觉测量技术及深度学习模型的飞速发展,三维视觉成为人工智能、虚拟现实等领域的重要支撑技术,三维人脸成像及重建技术取得了突破性进展,不仅能够更好地应对光照、遮挡、表情和姿态等变化,同时增大了伪造攻击难度,大大推动了真实感“虚拟数字人”的重建与渲染,有效提升了人脸系统的安全性。本文对三维人脸成像技术和重建模型进行了全面综述,尤其对基于深度学习的三维人脸重建进行系统深入地分析。

Introduction

This review provides a comprehensive roadmap of the last five years in 3D facial technology, tracing the evolution from expensive, specialized hardware to sophisticated deep learning models. While 2D facial recognition and analysis are ubiquitous, they struggle with "in-the-wild" challenges such as dramatic lighting changes, extreme head poses, and occlusions.

3D facial data offers a robust solution by capturing the actual geometry and material properties of the face, making systems more secure against spoofing. The current "holy grail" of the field is achieving high-fidelity 3D reconstruction from limited input—such as a single 2D smartphone photo—without requiring laboratory-grade equipment. This review synthesizes how researchers are closing the gap between low-quality input and high-quality 3D output.

Methodology and Taxonomy

The authors categorize the technology into two main pillars: Imaging Systems (the hardware) and Reconstruction Models (the software). The following diagram outlines the systematic approach taken in this review to categorize these technologies.

Taxonomy of 3D face imaging and reconstruction technologies.

1. Imaging Technology

Imaging systems have evolved from complex laboratory setups to consumer-level active sensors.

  • Active Sensing: Uses light projection (Structured Light or Time-of-Flight) to measure depth. Examples include the Microsoft Kinect and the iPhone’s FaceID.
  • Passive Sensing: Relies on standard cameras and algorithms like Multi-View Stereo (MVS).
  • Hybrid Systems: High-end setups like the Light Stage or Medusa system use hundreds of lights and cameras to capture "ground truth" data, including sub-millimeter skin pores and complex light reflectance (BRDF).
High-end imaging setups like the Light Stage used for capturing high-fidelity facial data.

2. Reconstruction Algorithms

The paper tracks the shift from Linear Models (like the classic 3D Morphable Model or 3DMM) to Non-linear Deep Learning Models:

  • CNNs & GANs: Used to regress 3D parameters from 2D images. StyleGAN-based methods are particularly noted for generating realistic skin textures.
  • Implicit Neural Representations (INR): Instead of a mesh of triangles, the face is represented as a continuous mathematical function, allowing for "infinite" resolution and better handling of fine details like wrinkles.
  • Neural Radiance Fields (NeRF): The current frontier. NeRFs allow for photorealistic rendering from any angle by learning a volumetric representation of the head.

Datasets and Benchmarks

Deep learning progress has been fueled by the transition from small, high-quality datasets (BU-3DFE) to large-scale 3D datasets like FaceScape (938 people, 18k+ samples) and FaceVerse. For evaluation, the NoW (Next on Web) and REALY datasets are highlighted as the current gold standards for measuring performance on real-world, unconstrained images.

Key Insights and Results

  • The Death of the "Average" Face: Modern deep learning (INR and NeRF) has broken the bottleneck of traditional 3DMMs, which could only represent faces as linear combinations of "average" shapes.
  • Self-Supervision: The most successful recent models use "analysis-by-synthesis," allowing them to learn from millions of unlabelled 2D internet photos by guessing 3D shapes and rendering them back to 2D for comparison.
  • Quantitative Performance: Models like HRN (Hierarchical Representation Network) and Deep3D currently lead in accuracy, achieving Mean Square Error (MSE) scores as low as 1.53mm on the REALY benchmark.

Limitations and Future Work

Despite significant progress, several challenges remain:

  1. Extreme Conditions: Models still struggle when the head is turned more than 60 degrees or heavily occluded.
  2. The "Uncanny Valley" in Motion: Capturing the subtle, non-linear dynamics of speech and emotion remains difficult.
  3. Mobile Efficiency: Shrinking high-end NeRF and GAN models for real-time use on smartphones is a major area of active research.
  4. Audio-Visual Synergy: Better integration of audio cues to drive 3D facial animation is needed for more convincing virtual assistants.

Summary updated

Abstract

得益于新型三维视觉测量技术及深度学习模型的飞速发展,三维视觉成为人工智能、虚拟现实等领域的重要支撑技术,三维人脸成像及重建技术取得了突破性进展,不仅能够更好地应对光照、遮挡、表情和姿态等变化,同时增大了伪造攻击难度,大大推动了真实感“虚拟数字人”的重建与渲染,有效提升了人脸系统的安全性。本文对三维人脸成像技术和重建模型进行了全面综述,尤其对基于深度学习的三维人脸重建进行系统深入地分析。首先,对三维人脸成像设备及采集系统进行详细梳理及对比归纳,并介绍了基于新传感技术的人脸成像系统;然后,对基于深度学习的三维人脸重建模型进行系统分析,从输入数据源角度分为基于单目图像、基于多目图像、基于视频和基于语音的三维人脸重建算法4类。通过深入分析,总结三维人脸成像的研究现状及面临的难点与挑战,对未来发展方向及应用进行积极探讨与展望。本文涵盖了近5年经典的三维人脸成像及重建相关的技术与研究,为人脸研究、发展和应用提供了很好的参考。
BibTeX
@article{liu2024d,
  author = {Fei Liu and Kunbo Zhang and Qing Yang and Shubo Zhou and Yunlong Wang and Zhenan Sun},
  title = {{3D face imaging and reconstruction technology: a review}},
  journal = {Journal of Image and Graphics},
  year = {2024},
  doi = {10.11834/jig.230697},
}
Figure