TL;DR
This paper tackles the challenge of accurately detecting 3D facial landmarks in real-time using depth data, which is often noisy. They introduce a new dataset and a novel deep learning approach that leverages auxiliary information (like 2D landmarks) to significantly improve the robustness and accuracy of 3D landmark prediction, outperforming methods relying solely on depth or visual data.
What Problem Does This Paper Address?
Modern facial motion capture systems often rely on visual (2D) data for feature tracking and expression prediction, while depth (3D) data is used to build 3D face models. However, using a single data stream can limit the understanding of 3D facial structure. While depth sensors have become more affordable and performant, their inherent noise can hinder accurate facial landmark detection. Existing research often focuses on either 2D landmark prediction or uses visual data, with less exploration into leveraging depth data effectively, especially for 3D landmark prediction. This paper aims to bridge this gap by investigating how different data streams (RGB, grayscale, depth, and combinations) and auxiliary information can improve real-time 3D facial landmark detection.
Methods
The core of this work involves using Convolutional Neural Networks (CNNs) for facial landmark detection. The researchers experimented with several network architectures:
- Single-Stream Networks: These networks take a single type of input (e.g., grayscale, RGB, or depth) and process it through convolutional layers to extract features, followed by fully connected layers to predict landmarks.
- Merged Networks: These networks combine multiple data streams (e.g., RGBD or GSD). They employ a multi-input model where separate CNNs process each data stream, and their outputs are then merged and fed into a third network for further processing.
- Auxiliary Information: A key innovation is the use of auxiliary information. The networks are trained to predict not only 2D (UV) and 3D (XYZ) coordinates but also a combined UVXYZ output. This allows the network to learn from both appearance-based features (UV) and geometric structure (XYZ), with the latter being particularly challenging for depth data alone.
The researchers also introduced a novel dataset, the Kinect One Expression Dataset (KOED), to address data deficiencies in this domain.
Datasets
- Kinect One Expression Dataset (KOED): This is a newly created in-house dataset designed to overcome existing data limitations. It features 35 participants performing seven universal facial expressions (Happy, Sad, Surprise, Anger, Fear, Contempt, Disgust) with synchronized RGB and Depth data captured using a Kinect One sensor at 30 fps.
- Existing Datasets (Reviewed but not used for training): The paper reviews several existing datasets like Face Warehouse, Biwi Kinect Head Pose, Eurocom Kinect, VAP face database, and 3D Mask Attack. However, these were found to be unsuitable for the project's requirements due to limitations in scale, resolution, lack of 3D landmarks, or reliance on older Kinect versions.
Key Insights and Ideas
- Depth Data is Noisy but Valuable: While depth data alone struggles with accuracy due to noise, when combined with visual data (RGB or grayscale), it significantly improves 3D landmark prediction.
- Auxiliary Information is Crucial for 3D: Predicting 3D (XYZ) landmarks is inherently more difficult than 2D (UV). The introduction of auxiliary information, particularly by training for combined UVXYZ outputs, helps the network learn more robust geometric representations and standardize 3D locations.
- Grayscale (Gs) Can Be Surprisingly Effective: For certain tasks, especially when combined with depth (GSD), grayscale data can perform comparably to or even better than RGB, suggesting that a clean visual stream can be more important than color for some landmarking tasks.
- Single-Stream Depth is Challenging: Networks relying solely on depth data exhibit unstable learning curves and the worst performance, highlighting the need for visual assistance to overcome noise and accurately locate landmarks.
Key Results
- RGB Outperforms for UV: For 2D (UV) landmark prediction, RGB data achieved the lowest Mean Squared Error (MSE) and Mean Absolute Error (MAE).
- Merged Data Excels for 3D: For 3D (XYZ) landmark prediction, merged data streams (RGBD and GSD) provided the highest precision, scoring the lowest MSE and MAE.
- Auxiliary Information Improves UVXYZ: In the combined UVXYZ prediction, the Gs network performed best overall. The introduction of 3D landmarks significantly reduced the overall loss compared to UV-only prediction, and also improved XYZ prediction.
- Gs is Robust: The Gs network, particularly when combined with depth (GSD), showed strong performance in feature map representation, indicating its ability to focus on specific landmark regions and process data efficiently.
- Real-time Potential: The proposed methods are designed for real-time performance, with networks trained to operate at 30 frames per second.
Limitations and Future Work
- Depth Noise: Despite improvements, depth data still presents noise, which can lead to precise Z-position inaccuracies in some cases.
- Controlled Environment: The KOED dataset was captured in a controlled environment with cropped depth maps. This limits the network's exposure to the full 3D viewing frustum, which might be necessary for more general 3D prediction.
- Dataset Size and Diversity: While KOED is an improvement, larger datasets with wider demographic representation would further enhance reliability.
- Future Work: The authors suggest exploring newer depth sensing technologies (like Intel RealSense) for higher resolution and cleaner depth maps. They also propose expanding the dataset and increasing the number of reliably tracked landmarks. Further research could also investigate the impact of different auxiliary features and explore more advanced network architectures.
Summary updated