3D Face Recognition Based on Multiple Keypoint Descriptors and Sparse Representation

Lin Zhang, Zhixuan Ding, Hongyu Li, Ying Shen, Jianwei LuPLoS 2014 · DOI

Abstract

Recent years have witnessed a growing interest in developing methods for 3D face recognition. However, 3D scans often suffer from the problems of missing parts, large facial expressions, and occlusions. To be useful in real-world applications, a 3D face recognition approach should be able to handle these challenges. In this paper, we propose a novel general approach to deal with the 3D face recognition problem by making use of multiple keypoint descriptors (MKD) and the sparse representation-based classification (SRC). We call the proposed method 3DMKDSRC for short. Specifically, with 3DMKDSRC, each 3D face scan is represented as a set of descriptor vectors extracted from keypoints by meshSIFT. Descriptor vectors of gallery samples form the gallery dictionary. Given a probe 3D face scan, its descriptors are extracted at first and then its identity can be determined by using a multitask SRC. The proposed 3DMKDSRC approach does not require the pre-alignment between two face scans and is quite robust to the problems of missing data, occlusions and expressions.

Introduction

While 3D facial scans offer significantly more geometric detail than traditional 2D images, they present unique processing challenges. Most "holistic" 3D recognition methods—which analyze the face as a single, unified surface—depend heavily on precise alignment (registration) via algorithms like Iterative Closest Point (ICP). These methods often fail when confronted with non-rigid deformations caused by facial expressions or when data is missing due to occlusions such as hair, glasses, or hands.

To address these limitations, we introduce 3DMKDSRC (3D Multiple Keypoint Descriptors and Sparse Representation-based Classification). This framework moves away from global alignment and landmark-dependent measurements, instead leveraging a dense set of local descriptors and a robust multi-task sparse coding scheme.

The 3DMKDSRC Methodology

The 3DMKDSRC pipeline is divided into two primary stages: local feature extraction and sparse classification.

1. Feature Extraction via meshSIFT

The system treats the 3D mesh as a scale-space, identifying "keypoints" at various scales by searching for extrema in mean curvature. This approach ensures that the features captured are stable across different resolutions and orientations.

Scale-space variations in 3D face scans across different subjects and smoothing parameters.

For each detected keypoint, a descriptor vector is calculated based on two geometric properties:

  • Shape Index: This characterizes the local "type" of the surface (e.g., spherical cap, ridge, or rut).
  • Slant Angles: These represent the orientation of the surface normal across nine local sub-regions surrounding the keypoint.
Keypoints detected on a 3D face scan using the meshSIFT algorithm.

By pooling these descriptors, the face is represented as a collection of local geometric "signatures" rather than a single rigid shape. This allows the system to remain robust even if parts of the face are missing or distorted.

2. Multi-task Sparse Representation-based Classification (SRC)

Once features are extracted, the recognition problem is framed as a sparse reconstruction task. All descriptors from the "gallery" (the database of known individuals) are compiled into a large dictionary. For an unknown "probe" scan, the algorithm attempts to represent each of its descriptors as a sparse linear combination of the gallery descriptors.

To maintain efficiency and accuracy, the system employs "Dictionary Shrinking" to pre-select likely matches. Furthermore, a Sparsity Criterion is used to filter out non-discriminative keypoints—those that match many subjects equally poorly—ensuring that only high-quality, unique features contribute to the final identity decision.

Robustness and Key Insights

A defining characteristic of 3DMKDSRC is its alignment-free nature. By relying on local descriptors, the system bypasses the computationally expensive and error-prone ICP registration step.

Furthermore, the method relies on a consensus of keypoints rather than a few specific landmarks like the nose tip or eye corners. If a specific region of the face is occluded or distorted by an extreme expression, the hundreds of other stable keypoints (e.g., on the forehead or chin) provide sufficient information for accurate identification.

Experimental Evaluation

The 3DMKDSRC method was rigorously tested on three benchmark databases:

  • Bosphorus: Known for its variety of expressions and occlusions, the method achieved a 95.03% Rank-1 recognition rate.
  • GavabDB: Despite the high noise levels in this dataset, the system reached 100% accuracy on neutral scans and 92.62% overall.
  • FRGC2.0: On this large-scale standard benchmark, 3DMKDSRC achieved 89.29% accuracy, outperforming the baseline meshSIFT without requiring any 3D preprocessing like hole-filling.

Conclusion

3DMKDSRC provides a robust, general-purpose solution for 3D face recognition that excels in real-world conditions involving expressions and occlusions. While the computational cost of $l_1$-minimization remains a consideration for real-time applications, the method's ability to function without complex pre-alignment or landmark detection marks a significant advancement in the field.

Summary updated

Abstract

Recent years have witnessed a growing interest in developing methods for 3D face recognition. However, 3D scans often suffer from the problems of missing parts, large facial expressions, and occlusions. To be useful in real-world applications, a 3D face recognition approach should be able to handle these challenges. In this paper, we propose a novel general approach to deal with the 3D face recognition problem by making use of multiple keypoint descriptors (MKD) and the sparse representation-based classification (SRC). We call the proposed method 3DMKDSRC for short. Specifically, with 3DMKDSRC, each 3D face scan is represented as a set of descriptor vectors extracted from keypoints by meshSIFT. Descriptor vectors of gallery samples form the gallery dictionary. Given a probe 3D face scan, its descriptors are extracted at first and then its identity can be determined by using a multitask SRC. The proposed 3DMKDSRC approach does not require the pre-alignment between two face scans and is quite robust to the problems of missing data, occlusions and expressions. Its superiority over the other leading 3D face recognition schemes has been corroborated by extensive experiments conducted on three benchmark databases, Bosphorus, GavabDB, and FRGC2.0. The Matlab source code for 3DMKDSRC and the related evaluation results are publicly available at http://sse.tongji.edu.cn/linzhang/3dmkdsrcface/3dmkdsrc.htm.
BibTeX
@inproceedings{zhang2014d,
  author = {Lin Zhang and Zhixuan Ding and Hongyu Li and Ying Shen and Jianwei Lu},
  title = {{3D Face Recognition Based on Multiple Keypoint Descriptors and Sparse Representation}},
  booktitle = {PLoS},
  year = {2014},
  doi = {10.1371/journal.pone.0100120},
}
Figure