Few-data guided learning upon end-to-end point cloud network for 3D face recognition

Yi Yu, Feipeng Da, Ziyu Zhang

TL;DR

The authors present Sur3dNet-Face, an end-to-end deep learning network that performs 3D face recognition directly on point clouds. By combining a modified PointNet architecture with a novel training strategy that uses synthetic data guided by a tiny amount of real data (only 943 scans), they achieve state-of-the-art accuracy without the massive datasets typically required by deep learning.

What Problem Does This Paper Address?

Deep learning has revolutionized face recognition, but it is notoriously "data-hungry." In the 3D domain, high-quality scans are much harder to come by than 2D images. Many current "state-of-the-art" models are trained on tens of thousands of scans, which makes it difficult to tell if their performance comes from a superior architecture or simply a larger training set. Furthermore, in many practical applications, the available training data is much smaller than the eventual test set.

The authors tackle two main challenges:

  1. Data Scarcity: How to train a robust deep network when you only have a few hundred real 3D scans.
  2. Point Cloud Limitations: Standard point cloud architectures like PointNet often struggle with face recognition because they tend to sample "unstable" edge points or lose crucial physical scale information during normalization.

Methods

The researchers introduced several key innovations to make PointNet work for 3D faces:

  • Sur3dNet-Face Architecture: An end-to-end network that takes raw point cloud coordinates as input. Unlike many methods that project 3D data into 2D "depth maps," this stays in 3D space to maintain rotation and transformation invariance.
  • Dithering Farthest Point Sampling (DFPS): Standard Farthest Point Sampling (FPS) often picks points at the boundaries of a scan, which are noisy and vary wildly with pose. DFPS introduces a "valid radius" to focus on the central facial features and uses surface curvature (eigenvalues) to "dither" the sampling, making the network more robust to noise.
  • Physical Scale Preservation: Most networks normalize coordinates to a $(-1, 1)$ range. Sur3dNet-Face keeps the original millimeter scale, recognizing that the actual size of a face is a vital biometric feature.
  • Few-Data Guided Learning: To train the model, they used a Gaussian Process Morphable Model (GPMM) to generate thousands of synthetic identities. Crucially, they didn't just generate random faces; they used the small amount of real data they had to "guide" the synthetic expressions, ensuring the generated data matched the distribution of real-world facial movements.
  • Real-Data Verification: During training, the model is validated against a small set of real data to ensure it isn't just "memorizing" the quirks of the synthetic generator.

Datasets

  • FRGC v2.0: The "Spring2003" subset (943 scans) was used for training guidance and verification. The rest was used for testing.
  • Bosphorus Dataset: Used as a cross-dataset benchmark to evaluate the model's ability to generalize to different scanning conditions and populations.

Key Insights and Ideas

  • The "Edge" Problem: The authors realized that PointNet’s default sampling logic is its Achilles' heel for faces. By forcing the network to ignore the unstable edges of a 3D scan and focus on the high-entropy central regions (nose, eyes, mouth), they significantly boosted stability.
  • Synthetic Data Needs a Compass: Purely synthetic data often lacks the nuance of real human expressions. By using real scans to calculate "expression coefficients" and applying those to synthetic identities, they created a "best of both worlds" training set: the volume of synthetic data with the realism of real data.
  • Scale is a Feature: In 3D vision, we often throw away scale to achieve "scale invariance." This paper argues that for faces, scale is a feature, not a bug. Keeping the millimeter units helps distinguish between individuals who might have similar facial proportions but different absolute sizes.

Key Results

Despite being trained on a fraction of the data used by competitors, Sur3dNet-Face achieved impressive results:

  • Rank-1 Recognition Rate (RR1): 98.85% on FRGC v2.0 and 99.33% on the Bosphorus dataset.
  • Generalization: The model performed exceptionally well on the Bosphorus dataset without any fine-tuning, proving that the GPMM-based training generalizes well to unseen data.
  • Efficiency: The system is fast, taking only 0.035s for preprocessing (nose detection and normal calculation) and 0.105s for feature extraction on a single GPU.

Limitations and Future Work

  • Extreme Poses and Occlusions: The current sampling strategy (DFPS) is optimized for relatively frontal scans. The authors excluded the "occlusion" and "large posture" subsets of the Bosphorus dataset from their primary results.
  • Future Directions: The authors plan to optimize the data generation process to include more "angled" faces and simulated occlusions (like hands or glasses) to improve the model's robustness in unconstrained, real-world environments.

Summary updated

Abstract

Deep-learning-based 3D face recognition methods have developed vigorously in recent years, while the potential of these methods is being exploited in more and more scenarios. In this paper, an end-to-end deep learning network entitled Sur3dNet-Face for point-cloud-based 3D face recognition is proposed. The method uses PointNet, which is a successful point cloud classification solution but performs unexpectedly in face recognition, as the backbone. To adapt the backbone to 3D face recognition, modifications in network architecture and a few-data guided learning framework based on Gaussian process morphable model is supplemented. Instead of mass data in multiple datasets for training, our method takes only Spring2003 subset of FRGC v2.0 for training which contains 943 facial scans and the network is well trained with such a small amount of real data. The processing time to generate face representation is less than 0.15 s. Without fine-tuning on the test set, the Rank-1 Recognition Rate (RR1) is achieved as follows: 98.85% on FRGC v2.0 dataset and 99.33% on Bosphorus dataset, which proves the effectiveness and the potentiality of our method. When facing scenarios with limited resource, the proposed method is expected to give a competitive performance.
BibTeX
@article{yu2022few,
  author = {Yi Yu and Feipeng Da and Ziyu Zhang},
  title = {{Few-data guided learning upon end-to-end point cloud network for 3D face recognition}},
  journal = {Multimedia Tools and Applications},
  year = {2022},
  url = {https://openreview.net/forum?id=0hZt38Z4vDI},
}

Loading related content...

Figure