Highly Accurate and Fully Automatic 3D Head Pose Estimation and Eye Gaze Estimation Using RGB-D Sensors and 3D Morphable Models

Reza Shoja Ghiass, Ognjen Arandjelovic, Denis LaurendeauSensors 2018 · OpenReview

TL;DR

This paper presents a fully automatic and highly accurate 3D head pose and eye gaze estimation system using low-quality RGB-D sensors. It overcomes limitations of previous methods by modeling only the face (not the entire head) and avoiding manual intervention or offline training, making it more practical for real-world applications.

What Problem Does This Paper Address?

Estimating a person's head pose (orientation) and gaze direction is crucial for understanding human behavior and enabling various applications, from human-computer interaction to driver monitoring. Traditional methods often rely on high-resolution images, require significant subject cooperation (e.g., specific head movements), or involve complex offline model construction. Consumer-grade RGB-D sensors like the Microsoft Kinect offer depth information, but their lower quality presents challenges for precise estimation. This paper aims to bridge this gap by developing a robust, automatic, and non-intrusive system that can accurately estimate head pose and gaze using these affordable sensors, even with variations in head pose and partial occlusions.

Methods

The core of the proposed method involves a novel approach to 3D head pose estimation and its subsequent application to gaze estimation.

  • Subject-Specific Face Modeling: Instead of reconstructing the entire head, the system focuses on modeling only the subject's face. This is achieved by:
  • Geometry Processing: Initial depth data is processed to remove outliers and isolate the facial region. This involves detecting facial features (eyes, nose, mouth) on the RGB frame and using them to define a cropping plane for the depth data.
  • Generic Model Positioning: A pre-existing 3D Morphable Model (3DMM) of a generic face is aligned with the trimmed depth data of the subject's face using a rigid Iterative Closest Point (ICP) algorithm.
  • Face Morphing: The generic 3DMM is then "morphed" to fit the specific subject's face by finding optimal weights for the model's principal components. This process is framed as a cost function minimization problem, unifying ICP and PCA terms.
  • Head Pose Tracking: Once the subject-specific face model is constructed, the system tracks the head pose in subsequent depth frames. This is done using a robust ICP formulation that is less sensitive to outliers and partial occlusions, leveraging Tukey functions to down-weight unreliable correspondences. The head pose (yaw, pitch, roll) is directly derived from the rotation matrix obtained during registration.
  • Gaze Estimation: The estimated head pose is then used for gaze estimation. The system employs an appearance-based approach, meaning it learns a mapping from the appearance of the eye region to the gaze vector.
  • Pose-Free Eye Appearance: The system first extracts eye regions from the frontal view of the subject's model.
  • Training: An interpolation function is trained to map these pose-free eye appearances to gaze vectors. Two methods are explored: Adaptive Linear Regression (ALR) and K-Nearest Neighbors (KNN). The gaze vectors for training are obtained using a calibration pattern.
  • Testing: During testing, the system estimates the Point of Regard (PoR) by calculating the line of sight in the world coordinate system and intersecting it with the display screen.

Datasets

The paper utilizes two publicly available datasets for evaluation:

  • Biwi Head Pose Database: This RGB-D dataset, containing over 15,000 depth frames and RGB images from 20 subjects, was used for evaluating the head pose estimation accuracy. It provides ground truth for head rotation.
  • EYEDIAP Database: This dataset, comprising over 4450 depth frames and RGB images from 16 subjects, was used for evaluating the gaze estimation performance. It includes scenarios with both stationary and moving heads.

Key Insights and Ideas

  • Face-Centric Modeling is Sufficient: The key insight is that modeling only the face, rather than the entire head, is sufficient for accurate pose and gaze estimation with RGB-D sensors. This simplifies the problem and relaxes constraints imposed by full head models.
  • Robustness through Tukey Functions: The use of Tukey functions in the ICP algorithm significantly improves robustness to outliers and partial occlusions, which are common issues with low-quality depth data.
  • Automatic and Non-Intrusive: The system's ability to be fully automatic, requiring no manual intervention or subject cooperation (like chin rests or specific head movements), is a major advancement for practical usability.
  • Appearance-Based Gaze with Precise Pose: By combining a highly accurate head pose tracker with appearance-based gaze estimation, the system achieves better gaze estimation results than previous appearance-based methods, which often struggle with head pose variations.

Key Results

  • Head Pose Estimation: The proposed method achieved competitive accuracy on the Biwi head pose database, demonstrating its effectiveness in handling large pose variations and partial occlusions. While specific error metrics are provided in Table 1, the paper highlights that their system shows better average error than Fanelli et al.'s work [2,3].
  • Gaze Estimation: The system achieved promising results on the EYEDIAP database for both stationary and moving head scenarios. Tables 10-13 show detailed error metrics for ALR and KNN methods. For instance, in stationary head scenarios, the proposed method achieved gaze estimation errors of 7.55° (left eye) and 6.89° (right eye) using ALR, outperforming Funes and Odobez [7,34]. The paper attributes this to the higher precision of their pose estimation system.

Limitations and Future Work

  • Limitations:
    • The system's tracking mode requires the subject's head motion in successive frames to be relatively small. It cannot start from an extreme pose.
  • While robust to some facial expression changes, extreme expressions might still pose challenges.
  • The accuracy of the gaze estimation is dependent on the precision of the head pose estimation.
  • Future Work:
    • The paper implicitly suggests further improvements in handling more extreme head poses and faster motions.
  • Exploring more advanced gaze estimation techniques that can further leverage the precise head pose information could be a direction.
  • Investigating the system's performance with even lower quality depth sensors or in more challenging real-world environments.

Summary updated

Abstract

This work addresses the problem of automatic head pose estimation and its application in 3D gaze estimation using low quality RGB-D sensors without any subject cooperation or manual intervention. The previous works on 3D head pose estimation using RGB-D sensors require either an offline step for supervised learning or 3D head model construction, which may require manual intervention or subject cooperation for complete head model reconstruction. In this paper, we propose a 3D pose estimator based on low quality depth data, which is not limited by any of the aforementioned steps. Instead, the proposed technique relies on modeling the subject’s face in 3D rather than the complete head, which, in turn, relaxes all of the constraints in the previous works. The proposed method is robust, highly accurate and fully automatic. Moreover, it does not need any offline step. Unlike some of the previous works, the method only uses depth data for pose estimation. The experimental results on the Biwi head pose database confirm the efficiency of our algorithm in handling large pose variations and partial occlusion. We also evaluated the performance of our algorithm on IDIAP database for 3D head pose and eye gaze estimation.
BibTeX
@inproceedings{ghiass2018highly,
  author = {Reza Shoja Ghiass and Ognjen Arandjelovic and Denis Laurendeau},
  title = {{Highly Accurate and Fully Automatic 3D Head Pose Estimation and Eye Gaze Estimation Using RGB-D Sensors and 3D Morphable Models}},
  booktitle = {Sensors},
  year = {2018},
  url = {https://openreview.net/forum?id=7JPaUp8uhX},
}
Figure