Fine Gaze Redirection Learning With Gaze Hardness-Aware Transformation

Sangjin Park, Daeha Kim, Byung Cheol SongWACV 2023

TL;DR

This paper introduces a novel method for "fine gaze redirection," which aims to precisely adjust a person's gaze direction in an image. They propose a "Gaze Hardness-aware Transformation" (GHT) to generate diverse gaze features and a "structured gaze (SG) loss function" to disentangle gaze features from irrelevant ones like head pose. This approach achieves state-of-the-art results, improving redirection error by 10% on the GazeCapture dataset.

What Problem Does This Paper Address?

Gaze redirection, the task of altering a person's gaze direction in an image, has applications in areas like human-computer interaction and virtual reality. While previous methods have shown success, they often struggle with learning fine-grained gaze directions and disentangling gaze information from other facial attributes like head pose and appearance. Existing approaches tend to learn from a limited set of gaze directions present in the training data, making it difficult to generalize to unseen directions. Furthermore, when gaze features are entangled with head pose or appearance features in the model's latent space, it becomes challenging to learn a representation that solely captures gaze. This paper tackles these limitations by proposing methods to learn more robust and disentangled gaze representations.

Methods

The paper introduces two main novel components:

  • Gaze Hardness-aware Transformation (GHT): This technique generates new gaze features that represent diverse gaze directions, going beyond those present in the source and target images. GHT achieves this through linear interpolation of source and target gaze features in the latent space. This interpolated feature, z_tr, acts as an additional supervision signal, increasing the variety of gaze directions the model learns from and making the learning process more challenging, thus preventing trivial solutions and overfitting. A "gaze consistency loss" (L_lens) is used to ensure these transformed features align with the target gaze direction.
  • Structured Gaze (SG) Loss Function: This is a metric learning approach designed for discriminative learning of gaze features. It aims to disentangle gaze features from inessential features like head pose (z_h) and appearance (z_u). The SG loss defines negative pairs between gaze features and these inessential attributes. To create more robust negative samples, it uses linear interpolation of these inessential attributes. For positive pairs, it leverages fine-grained eye features extracted from a separate pre-trained encoder. To prevent trivial solutions in learning positive pairs, it employs "feature extrapolation" to generate hard positive samples. The SG loss considers all combinations of positive and negative pairs within a mini-batch, forming a structured similarity learning approach.

The overall model architecture is based on a transforming auto-encoder (TA) framework, where an encoder E extracts features and a generator G reconstructs the image. A pre-trained network ψ is used for estimating gaze and head pose from the generated images for calculating the redirection loss (L_red).

Datasets

The paper uses the following datasets:

  • GazeCapture [20]: Used for training and within-dataset evaluation. It's a large dataset (2M images) collected in unconstrained settings.
  • MPIIGaze [43]: Used for within-dataset and cross-dataset evaluation. It contains images from 15 subjects in daily life.
  • Columbia Gaze [31]: Used for within-dataset and cross-dataset evaluation.
  • EYEDIAP [7]: Used for within-dataset and cross-dataset evaluation.

The authors note that informed consent was obtained for the EYEDIAP dataset.

Key Insights and Ideas

  • Feature-level interpolation for diverse gaze: The idea of interpolating latent gaze features to create new, unseen gaze directions is a clever way to expand the training data's effective gaze diversity without needing more labeled images.
  • "Hardness-aware" learning for robustness: By making the learning process harder (e.g., through GHT and hard negative/positive samples in SG loss), the model is forced to learn more robust and disentangled representations.
  • Disentanglement is crucial: The paper strongly emphasizes that disentangling gaze from head pose and appearance is key to achieving fine-grained gaze redirection. The SG loss function directly addresses this.
  • Metric learning for inter-feature relationships: Applying deep metric learning principles to learn the relationships between gaze and non-gaze features provides a principled way to achieve disentanglement.

Key Results

  • State-of-the-art performance: The proposed method achieved a redirection error (errg) of 1.884° on the GazeCapture dataset, which is a 14.2% improvement over the previous state-of-the-art method, STED [45].
  • Improved disentanglement: The method showed significant improvements in disentanglement metrics, such as h→g (effect of head pose on gaze), outperforming STED by 6.7% on GazeCapture.
  • Cross-dataset generalization: The model demonstrated strong generalization capabilities, outperforming STED on MPIIGaze, Columbia, and EYEDIAP datasets in cross-dataset evaluation.
  • Qualitative improvements: User studies indicated that the proposed method's generated images were preferred by users over STED's, suggesting better visual quality and realism. Visualization techniques like ContraCAM confirmed that the model focuses on relevant eye regions for gaze and separates inessential features.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the SG loss function helps disentangle features, it still relies on a limited number of "negative attributes" (head pose and task-irrelevant features). The paper also mentions that the influence of z_tr (generated by GHT) on the learning process is controlled by an adaptive coefficient a_sim, which is learned.
  • Future Work: The authors suggest that GHT can be extended to other gaze representation learning tasks that require extensive annotation. They also propose that the SG loss function's discriminative learning approach could be applied to other computer vision tasks, such as facial emotion or gesture recognition. They also mention exploring other metric learning losses and the impact of batch size in future work.

Summary updated

BibTeX
@inproceedings{park2023fine,
  author = {Sangjin Park and Daeha Kim and Byung Cheol Song},
  title = {{Fine Gaze Redirection Learning With Gaze Hardness-Aware Transformation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure