Three-D Wide Faces (3DWF): Facial Landmark Detection and 3D Reconstruction over a New RGB-D Multi-Camera Dataset

Marcos Quintana, Sezer Karaoglu, Federico Alvarez, José Manuel Menéndez, Theo Gevers

TL;DR

This paper introduces the "Three-D Wide Faces" (3DWF) dataset, a novel multi-camera RGB-D collection designed to advance 3D facial analysis. They also propose a data augmentation technique for facial landmark detection and demonstrate its effectiveness on existing deep learning models, contributing to more robust and accurate facial understanding in 3D.

What Problem Does This Paper Address?

The field of computer vision has seen rapid advancements in both deep learning and 3D imaging. However, to fully leverage these technologies for facial analysis tasks like pose estimation, gender/age classification, and 3D reconstruction, there's a critical need for comprehensive, annotated 3D facial datasets. Existing datasets often lack sufficient diversity in poses, lighting, or resolution, or they rely on expensive, specialized equipment. This paper aims to bridge this gap by creating a new, accessible dataset and demonstrating its utility for key facial analysis challenges.

Methods

The core of this work lies in two main contributions:

  • 3DWF Dataset Creation: The authors developed an innovative multi-camera RGB-D setup using three low-cost Asus Xtion depth cameras. This system was optimized for distance, light source, and camera orientation to capture high-quality, synchronized RGB-D data. The dataset includes raw and registered 3D data, along with demographic information (age, gender) for 92 subjects. They also incorporated high-accuracy 3D scans from a Faro Freestyle 3D Laser Scanner as a reference.
  • Data Augmentation for Facial Landmark Detection: To address the data-hungry nature of deep learning models, they propose a novel data augmentation method. This technique leverages 3D face meshes from the 3DU dataset and uses raycasting to project them into 2D images, simulating various head poses. This allows for the generation of a large, augmented dataset for training facial landmark detection models.
  • 3D Reconstruction Pipeline: A complete pipeline for 3D face reconstruction is presented, involving RGB-D capture, registration (aligning data from multiple cameras), refinement using Iterative Closest Point (ICP), and noise filtering.
  • Head Pose Classification: The dataset's utility is further demonstrated by using it to train and validate head pose classification models, employing techniques like Linear Discriminant Analysis (LDA) and Gaussian Naive Bayes (GNB).

Datasets

  • 3DWF Dataset (New Benchmark): This is the primary contribution, featuring synchronized RGB-D streams and 3D point clouds from 92 subjects across multiple poses. It also includes demographic data (age, gender) and reference scans from a Faro Freestyle 3D Laser Scanner.
  • 3DU Dataset: Used for the proposed data augmentation method, providing 3D face meshes of 300 subjects.
  • AFLW (Annotated Facial Landmarks in the Wild): Used for evaluating the performance of facial landmark detection models after fine-tuning with their augmented data.

Key Insights and Ideas

  • Accessibility of 3D Facial Data: The use of low-cost RGB-D sensors demonstrates that high-quality 3D facial data can be acquired without prohibitively expensive equipment, democratizing research in this area.
  • Data Augmentation for Robustness: The raycasting-based data augmentation from 3D meshes proves effective in generating diverse facial poses, significantly improving the performance of facial landmark detection models, especially when fine-tuning pre-trained networks.
  • Integrated Pipeline for 3D Facial Analysis: The paper presents a cohesive workflow from data acquisition to 3D reconstruction and analysis, highlighting the interconnectedness of these tasks.
  • Dataset Validation through Multiple Tasks: The 3DWF dataset is validated not only for its own quality but also by its successful application in both facial landmark detection and head pose classification, showcasing its versatility.

Key Results

  • Facial Landmark Detection: The proposed data augmentation method, when combined with pre-training on the 3DU dataset and fine-tuning on AFLW, achieved the lowest error rates for facial landmark detection, outperforming other combinations. Specifically, the RCN pipeline trained with 3DU data showed the best performance.
  • Head Pose Classification: Using the 3DWF dataset, LDA and GNB classifiers achieved high testing accuracies (83% and 84% respectively) for head pose classification, validating the dataset's quality and the alignment of its captured data. The confusion matrix indicated that predicting straight-on poses with pitch variations was simpler than predicting poses with side views.
  • 3D Reconstruction Accuracy: The reconstructed 3D face clouds (ClFo) showed an average minimum distance of 16-23 mm compared to ground truth scans (ClHD), which is within acceptable error ranges for depth sensors.

Limitations and Future Work

  • Gender Imbalance: The dataset has a slight imbalance in gender distribution, with more male subjects than female.
  • Age Distribution: While the dataset covers a wide age range, the majority of subjects fall between 20 and 40 years old, reflecting the university demographic where it was collected.
  • Future Work: The authors suggest further refinement of learning techniques for head pose estimation and exploring other facial analysis tasks that can benefit from the 3DWF dataset. They also highlight the potential for using the 3D information to create more feasible and less costly 2D facial analysis methods.

Summary updated

Abstract

Latest advances of deep learning paradigm and 3D imaging systems have raised the necessity for more complete datasets that allow exploitation of facial features such as pose, gender or age. In our work, we propose a new facial dataset collected with an innovative RGB–D multi-camera setup whose optimization is presented and validated. 3DWF includes 3D raw and registered data collection for 92 persons from low-cost RGB–D sensing devices to commercial scanners with great accuracy. 3DWF provides a complete dataset with relevant and accurate visual information for different tasks related to facial properties such as face tracking or 3D face reconstruction by means of annotated density normalized 2K clouds and RGB–D streams. In addition, we validate the reliability of our proposal by an original data augmentation method from a massive set of face meshes for facial landmark detection in 2D domain, and by head pose classification through common Machine Learning techniques directed towards proving alignment of collected data.
BibTeX
@inproceedings{quintana2019three,
  author = {Marcos Quintana and Sezer Karaoglu and Federico Alvarez and José Manuel Menéndez and Theo Gevers},
  title = {{Three-D Wide Faces (3DWF): Facial Landmark Detection and 3D Reconstruction over a New RGB-D Multi-Camera Dataset}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=4Gu7Hmcwgc_},
}
Figure