Long-Term Head Pose Forecasting Conditioned on the Gaze-Guiding Prior

Shentong Mo, Miao XinWorkshops @ CVPR 2021 · DOI

Abstract

This paper introduces the Conditional Spatial-Temporal Variational Graph Auto-Encoder (CST-VGAE), a novel framework for long-term head pose forecasting. Unlike current methods that often produce unrealistic or over-smoothed results due to the "one-to-many" nature of future movements, CST-VGAE leverages human physiological priors—specifically, the tendency for gaze to shift before the head moves. By integrating a gaze-guiding prior with a structured Gaussian Mixture Model (GMM) latent space and temporal self-attention, the model achieves state-of-the-art performance on benchmarks like the BIWI and UPNA datasets, providing more accurate and realistic 3D head orientation predictions over long time horizons.

Introduction

Head pose estimation is a well-established field, but forecasting future poses remains a significant challenge. The primary difficulty lies in the "one-to-many" mapping: for any given current state, there are multiple plausible future trajectories. Deterministic regression models often fail by predicting a single "average" future, resulting in blurry or unrealistic outputs. While generative models like standard VAEs offer more diversity, they frequently lack the necessary constraints to maintain realism in long-term scenarios. This work proposes to constrain the forecasting space using physiological priors and structural graph representations.

Methods

The proposed CST-VGAE framework utilizes several key innovations to improve forecasting accuracy and realism.

Gaze-Guiding Prior

A central contribution is the use of gaze direction as a conditional input. Physiological research indicates that gaze shifts at time $t$ are strong predictors of head pose at time $t+l$. The model exploits this correlation to "steer" the generative process toward plausible futures. Empirical analysis shows that the Pearson Correlation Coefficient (PCC) between gaze and head pose peaks at specific frame intervals, justifying this predictive relationship.

Correlation between gaze and head pose across different frame intervals.

CST-VGAE Architecture

The architecture employs a Gaussian Mixture Model (GMM) latent space with three components corresponding to the three degrees of freedom: yaw, pitch, and roll. This structured approach allows for a more interpretable and effective latent representation than a single Gaussian. A temporal self-attention module captures long-range dependencies between past gaze and predicted movements, while self-supervised learning on future landmarks further refines the output.

Detailed architecture of the CST-VGAE framework.

Results

The model was evaluated on the BIWI Kinect Head Pose Database and the UPNA Head Pose Database for generalizability. On the BIWI dataset, CST-VGAE achieved a Mean Absolute Error (MAE) of 6.27 for 30-step-ahead forecasts, significantly outperforming baselines like GAZE-LSTM (9.12) and standard VGAE (9.21). Cross-dataset evaluations on UPNA confirmed the model's robustness. Ablation studies demonstrated that both the gaze-guiding prior and the GMM latent space are critical to the model's superior performance.

Conclusion

CST-VGAE demonstrates the power of incorporating physiological constraints into generative models for human motion forecasting. While the model shows some limitations in static scenarios and depends on the quality of upstream landmark detectors, its ability to produce realistic, long-term head pose predictions opens new possibilities for applications in driver assistance systems and immersive VR/AR environments.

Summary updated

BibTeX
@inproceedings{mo2021long,
  author = {Shentong Mo and Miao Xin},
  title = {{Long-Term Head Pose Forecasting Conditioned on the Gaze-Guiding Prior}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  doi = {10.1109/cvprw53098.2021.00253},
}
Figure