HDFD - A High Deformation Facial Dynamics Benchmark for Evaluation of Non-Rigid Surface Registration and Classification

Gareth Andrews, Sam Endean, Roberto Dyke, Yu-Kun Lai, Gwenno Ffrancon, Gary K. L. TamarXiv 2018 · OpenReview

TL;DR

This paper introduces HDFD, a novel 4D facial dataset featuring extreme non-isometric deformations (funny faces) and spoken Welsh phrases, addressing the lack of challenging benchmarks for non-rigid surface registration. The dataset, with manually annotated landmarks and fluency ratings, is used to evaluate existing registration techniques, revealing their limitations with highly elastic deformations.

What Problem Does This Paper Address?

Accurately aligning and classifying 3D shapes that deform, like human faces, is crucial for many applications in computer vision and graphics. However, existing datasets for evaluating these "non-rigid surface registration" techniques often lack sufficient detail, particularly for highly elastic (non-isometric) deformations and dynamic sequences. This gap hinders the development of robust algorithms capable of handling real-world scenarios, such as those encountered in AR/VR applications. The paper aims to fill this void by creating a challenging benchmark dataset that pushes the boundaries of current registration methods.

Methods

The core contribution lies in the creation and evaluation of the HDFD dataset. The authors collected 4D (3D + time) facial data from 21 participants. Two main types of sequences were captured:

  • Funny Faces: Participants were asked to make extreme, unconstrained, and asymmetrical facial deformations, including mouth opening/closing and tongue protrusion. This intentionally introduces significant non-isometric changes and potential topological shifts.
  • Spoken Welsh Phrases: Participants spoke ten Welsh phrases with varying levels of fluency, from unintelligible to native. This introduces a visual-audio component and allows for fluency classification.

The captured data was meticulously processed:

  • 3D Reconstruction: A 3dMD system was used to capture high-resolution 3D geometry and texture at 48 fps, synchronized with audio.
  • Landmarking: Manually placed 3D landmarks were created for a subset of sequences. For the funny faces, an extended iBug template with 73 points was used, including specific points for tongue movement. For the Welsh phrases, landmarks were provided for a subset of participants and phrases.
  • Fluency Annotation: A Welsh language expert rated the fluency of each participant's spoken phrases on a scale of 0-5.

The paper also proposes a methodology for evaluating registration techniques using this dataset, involving selecting key frames and measuring registration errors using landmark correspondences.

Datasets

The paper introduces the HDFD (High Deformation Facial Dynamics) dataset, which comprises two main parts:

  • Funny Faces Dataset: Features 4D sequences of participants exhibiting extreme, non-isometric facial deformations. This dataset is intended to challenge registration algorithms with highly elastic changes and topological variations.
  • Welsh Visual-Audio Dataset: Contains 4D sequences of participants speaking Welsh phrases with varying fluency levels. This dataset supports research in visual-audio analysis, speech recognition, and fluency classification.

Both datasets were captured from 21 participants and include manually annotated 3D facial landmarks. The Welsh dataset is further annotated with expert-rated fluency levels. The authors state that existing datasets often focus on near-isometric deformations and lack detailed dynamic annotations, making HDFD a novel contribution.

Key Insights and Ideas

  • The "Funny Face" as a Stress Test: The deliberate use of unconstrained, extreme facial expressions as a benchmark is a clever way to expose the limitations of registration algorithms that rely on assumptions of limited deformation or specific topological structures.
  • Minority Language Data is Crucial: The inclusion of spoken Welsh data highlights the underrepresentation of minority languages in existing visual-audio datasets and its importance for broader research in speech and language processing.
  • Fluency Correlation with Mouth Shape: The paper suggests a potential correlation between 3D mouth/lip shapes and Welsh language fluency, opening avenues for automatic fluency assessment.
  • Landmarks are Key for Evaluation: The meticulous manual landmarking is presented as essential for providing ground truth for quantitative evaluation of registration techniques.

Key Results

The paper evaluates three state-of-the-art non-rigid registration techniques (NICP, L11, L12) on the funny face dataset.

  • L11 and L12 Outperform NICP: The more advanced techniques (L11 and L12), which incorporate sophisticated regularization constraints, consistently showed significantly lower error rates compared to the baseline NICP method.
  • Significant Error Variance: The errors varied considerably across different funny face sequences, reflecting the unconstrained nature of the participants' expressions and the inherent difficulty of the task.
  • Limitations in Mouth and Tongue Deformation: Despite the advancements, all tested methods struggled to accurately model the complex deformations of the mouth and tongue. These regions exhibited higher error rates, indicating they remain a significant challenge for current registration algorithms.
  • Artefacts Observed: Some registration results produced visual artefacts, such as black anomalies, further demonstrating the difficulty in accurately reconstructing highly deformed surfaces.

Limitations and Future Work

  • Dataset Size and Coverage: While the dataset is substantial, the authors acknowledge that the funny face sequences were captured from only 20 participants, and landmark annotations were provided for only 10 participants. Future work could involve expanding these annotations.
  • Registration Challenges: The paper clearly demonstrates that current state-of-the-art registration techniques still struggle with highly elastic deformations and topological changes, particularly around the mouth and tongue. This suggests a need for new algorithms that can better handle these complex scenarios.
  • Fluency Classification: The Welsh fluency dataset provides a foundation for classification, but further research is needed to develop and evaluate models for this task.
  • Release of Dataset: The authors intend to release the dataset to the community, which will be crucial for driving future research in this area.

Summary updated

Abstract

Objects that undergo non-rigid deformation are common in the real world. A typical and challenging example is the human faces. While various techniques have been developed for deformable shape registration and classification, benchmarks with detailed labels and landmarks suitable for evaluating such techniques are still limited. In this paper, we present a novel facial dynamic dataset HDFD which addresses the gap of existing datasets, including 4D funny faces with substantial non-isometric deformation, and 4D visual-audio faces of spoken phrases in a minority language (Welsh). Both datasets are captured from 21 participants. The sequences are manually landmarked, with the spoken phrases further rated by a Welsh expert for level of fluency. These are useful for quantitative evaluation of both registration and classification tasks. We further develop a methodology to evaluate several recent non-rigid surface registration techniques, using our dynamic sequences as test cases. The study demonstrates the significance and usefulness of our new dataset --- a challenging benchmark dataset for future techniques.
BibTeX
@misc{andrews2018hdfd,
  author = {Gareth Andrews and Sam Endean and Roberto Dyke and Yu-Kun Lai and Gwenno Ffrancon and Gary K. L. Tam},
  title = {{HDFD - A High Deformation Facial Dynamics Benchmark for Evaluation of Non-Rigid Surface Registration and Classification}},
  year = {2018},
  eprint = {1807.03354},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1807.03354},
}
Figure