Multimodal Face-Pose Estimation With Multitask Manifold Deep Learning

Chaoqun Hong, Jun Yu, Jian Zhang, Xiongnan Jin, Kyong-Ho Lee

TL;DR

This paper introduces a novel deep learning framework, Multi-task Manifold Deep Learning (M²DL), for more accurate human face pose estimation. By combining manifold learning to capture complex feature relationships with multi-task learning to handle diverse data modalities, M²DL achieves state-of-the-art performance on challenging benchmark datasets.

What Problem Does This Paper Address?

Estimating human face pose (gaze direction or head posture) from 2D images is crucial for applications like human activity analysis and human-computer interaction. However, this task is challenging due to variations in background, face orientation, and visibility. Existing methods often struggle with complex feature representations and mapping them to accurate poses, especially when dealing with multiple data sources (like RGB and depth images) or different viewing angles. The paper aims to overcome these limitations by developing a more robust and descriptive approach to face pose estimation.

Methods

The core of the proposed method, M²DL, lies in two key innovations:

  • Manifold Regularized Convolutional Layers (MRCL): Traditional Convolutional Neural Networks (CNNs) often treat neurons independently. MRCL enhances CNNs by incorporating manifold regularization. This allows the network to learn the underlying structure and relationships among neurons, preserving locality and leading to better feature representations. Essentially, it helps the network understand that similar features should be represented similarly, even if they appear in different parts of the image or under different conditions.
  • Multi-task Learning (MTL) for Multi-modal Data: M²DL treats different data modalities (e.g., RGB images, depth images) or different views as separate tasks. By learning these tasks simultaneously, the model can leverage shared information and learn a more generalized mapping from image features to head poses. This approach is particularly effective for combining diverse data sources, as it allows the model to learn task-specific representations while also benefiting from commonalities across tasks.

The overall framework involves face extraction, feature description using MRCL-enhanced CNNs, and finally, multi-task learning to map these features to pose estimations.

Datasets

The paper evaluates M²DL on three benchmark datasets:

  • DPOSE: This dataset consists of sequences from 16 subjects, captured by four cameras with overlapping fields of view. It includes head pan, tilt, and roll measurements. For experiments, 500 frames per camera were used for training.
  • HPID (Head Pose Image Database): This dataset contains monocular face images of 15 people with variations in pan and tilt angles. It's designed for training and testing on known and unknown faces. The first series of images was used for training, and the second for testing.
  • BKHPD (Biwi Kinect Head Pose Database): This dataset includes RGB and depth images from 20 people, with annotations for 3D head location and rotation. RGB and depth images were treated as two modalities, with 20% of the data used for training.

Key Insights and Ideas

  • Manifold Learning for Feature Representation: The paper highlights that simply using standard CNN features might not be sufficient for complex tasks like face pose estimation. Incorporating manifold learning (MRCL) helps capture the intrinsic, non-linear structure of the data, leading to more discriminative features.
  • Multi-task Learning for Data Fusion: Treating different modalities or views as separate tasks within a multi-task learning framework is a powerful way to fuse information. This allows the model to learn robust representations that generalize better across different data types.
  • ReLU as the Optimal Activation Function: Experiments showed that the Rectified Linear Unit (ReLU) activation function consistently yielded the best performance across all tested datasets, aligning with findings in recent deep learning research.
  • LeastSparseTrace Loss Function: Among various multi-task loss functions tested, LeastSparseTrace demonstrated superior performance, suggesting that its combination of sparsity and low-rank constraints is particularly effective for this problem.

Key Results

M²DL consistently outperformed existing state-of-the-art methods across all three benchmark datasets (DPOSE, HPID, and BKHPD) in terms of recovery errors (measured in Pan Degree). For instance, on the DPOSE dataset, M²DL achieved significantly lower recovery errors compared to methods like Linear Regression (LR), Twin Gaussian Processes (TGP), Salient Facial Structures (SFS), and others. The paper also showed that the combination of manifold learning and multi-task learning (M²DL) was crucial for achieving these superior results, outperforming configurations that used only one of these techniques or neither.

Limitations and Future Work

  • Limitations: While the paper demonstrates strong performance, the authors acknowledge that the accuracy of face detection and tracking can influence the overall pose estimation results, especially on datasets where face bounding boxes are not readily provided. The paper also mentions that traditional methods often struggle with low-resolution or blurred images, which could still be a challenge for M²DL.
  • Future Work: The paper suggests that further improvements could be made by exploring more sophisticated multi-task learning strategies and potentially incorporating other modalities or contextual information. The authors also imply that the framework could be extended to other related computer vision tasks that benefit from multi-modal data fusion and manifold learning.

Summary updated

Abstract

Face-pose estimation aims at estimating the gazing direction with two-dimensional face images. It gives important communicative information and visual saliency. However, it is challenging because of lights, background, face orientations, and appearance visibility. Therefore, a descriptive representation of face images and mapping it to poses are critical. In this paper, we use multimodal data and propose a novel face-pose estimation framework named multitask manifold deep learning ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{M}^2\text{DL}$</tex-math></inline-formula> ). It is based on feature extraction with improved convolutional neural networks (CNNs) and multimodal mapping relationship with multitask learning. In the proposed CNNs, manifold regularized convolutional layers learn the relationship between outputs of neurons in a low-rank space. Besides, in the proposed mapping relationship learning method, different modals of face representations are naturally combined by applying multitask learning with incoherent sparse and low-rank learning with a least-squares loss. Experimental results on three challenging benchmark datasets demonstrate the performance of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{M}^2\text{DL}$</tex-math></inline-formula> .
BibTeX
@article{hong2019multimodal,
  author = {Chaoqun Hong and Jun Yu and Jian Zhang and Xiongnan Jin and Kyong-Ho Lee},
  title = {{Multimodal Face-Pose Estimation With Multitask Manifold Deep Learning}},
  journal = {IEEE Trans. Industrial Informatics},
  year = {2019},
  doi = {10.1109/tii.2018.2884211},
}
Figure