TL;DR
This paper tackles the challenge of limited 3D face data for training deep learning models by proposing a novel framework that reconstructs millions of 3D faces from readily available 2D images. This "2D-aided" approach significantly boosts 3D face identification accuracy, achieving state-of-the-art results on several benchmarks.
What Problem Does This Paper Address?
Deep learning has revolutionized 2D face recognition, but its application to 3D face recognition is hampered by a critical bottleneck: the scarcity of large-scale 3D face datasets. Acquiring millions of high-quality 3D face scans is prohibitively expensive and time-consuming for both academic and industrial research. While existing 3D face recognition methods often rely on handcrafted features, they struggle with computational efficiency. Current deep learning approaches for 3D face recognition either use limited 3D data augmentation or don't leverage the vast amount of easily accessible 2D face images for training. This paper aims to bridge this gap by proposing a method to generate a massive 3D face dataset from 2D images and use it to train robust 3D face identification models.
Methods
The proposed framework, termed "2D-aided deep 3D face identification," involves four key stages:
- 2D Face Recognition Pre-training: A deep convolutional neural network (DCNN), specifically Sphere20, is pre-trained on a massive 2D face dataset (VGGFace2) for standard 2D face recognition. This leverages existing large-scale 2D data to learn general facial features.
- 2D to 3D Face Reconstruction: Millions of 3D face scans are generated from the 2D face images using a deep learning-based 3D face reconstruction method called ExpNet. ExpNet is chosen for its landmark-free approach and efficiency in regressing 3DMM (3D Morphable Model) parameters.
- Normal Component Image (NCI) Generation: For each reconstructed 3D face, facial normal maps are computed. These maps, representing the orientation of the facial surface, are then decomposed into three components: NCIx, NCIy, and NCIz. The paper argues that these normal maps capture more discriminative geometric information than raw 3D coordinates.
- 3D Face Identification Training and Inference: The pre-trained DCNN is further trained in a second phase using the generated NCIs (NCIx, NCIy, NCIz) of the reconstructed 3D faces. During inference, the NCIs of a probe 3D face are fed into the trained DCNN, and the resulting feature vectors are compared with gallery feature vectors using Euclidean distance for identification. Late fusion of scores from the three NCI networks is employed for the final prediction.
Datasets
- Training (2D): VGGFace2 (3.31 million images, 9,131 identities) was used for the initial pre-training of the DCNN on 2D face images.
- Training (3D Reconstruction): The paper implies using a large-scale 2D face database (VGGFace2) to reconstruct millions of 3D faces.
- Evaluation:
- BU-3DFE: 2500 3D facial scans of 100 subjects, featuring six expressions at four intensity levels plus neutral.
- FRGC v2.0: 4007 textured 3D face scans of 466 subjects, captured under controlled lighting.
- Bosphorus: 4666 textured 3D face scans of 105 subjects, with neutral and six basic facial expressions.
Key Insights and Ideas
- Leveraging 2D Data for 3D Tasks: The core idea is that the abundance of 2D face data can be effectively utilized to overcome the scarcity of 3D face data in deep learning for 3D face recognition.
- Reconstructed 3D Faces are Discriminative: The paper demonstrates that 3D faces reconstructed from 2D images, when processed correctly, can provide sufficient discriminative information for identity recognition.
- Facial Normals as Discriminative Features: Using Normal Component Images (NCIs) of the reconstructed 3D faces proves to be more effective than using raw 3D coordinates, as they highlight geometric details crucial for identification.
- Two-Phase Training is Crucial: A two-phase training strategy, first on large-scale 2D data and then on reconstructed 3D data (NCIs), is key to achieving high performance.
Key Results
The proposed 2D-aided deep 3D face identification framework achieved state-of-the-art results on several public 3D face recognition benchmarks:
- BU-3DFE: Rank-1 score of 98.8% (compared to 80.3% for 2D-trained model).
- FRGC v2.0: Rank-1 score of 97.6% (compared to 90.2% for 2D-trained model).
- Bosphorus: Rank-1 score of 98.4% (compared to 98.0% for 2D-trained model).
- Merged Database: A combined rank-one score of 98.2% was achieved across all three datasets.
The paper also shows significant improvements in rank-one scores when using NCIs compared to models trained solely on 2D images, highlighting the effectiveness of the reconstructed 3D information.
Limitations and Future Work
- Quality of Reconstructed 3D Faces: The authors acknowledge that the quality of the reconstructed 3D faces from ExpNet can limit identification accuracy, especially for fine details.
- ExpNet as the Sole Reconstruction Method: The framework currently relies on ExpNet for 3D reconstruction. Future work could explore other 3D face reconstruction methods.
- Recognition-Driven Reconstruction: The authors suggest investigating "recognition-driven 3D face reconstruction" methods, where the reconstruction process is optimized to produce faces that are more favorable for identification tasks. This could involve fine-tuning the reconstruction model based on recognition performance.
Summary updated