Rotation-Constrained Cross-View Feature Fusion for Multi-View Appearance-Based Gaze Estimation

Yoichiro Hisadome, Tianyi Wu, Jiawei Qin, Yusuke SuganoWACV 2024

TL;DR

This paper introduces a novel multi-view gaze estimation approach that leverages the relative rotation between cameras to fuse features adaptively. This method significantly improves generalization to unseen head poses, a key challenge in appearance-based gaze estimation, without substantial computational overhead.

What Problem Does This Paper Address?

Estimating a person's gaze direction from images is crucial for applications ranging from human-computer interaction to accessibility. Appearance-based methods, which use machine learning to infer gaze from facial or eye images, have shown promise but struggle with generalization, particularly when faced with unseen head poses. This is because head pose directly alters the visual patterns in the image, making it difficult for models trained on specific poses to perform well on new ones. While multi-view approaches can offer more robust information, training models for every possible camera configuration is impractical. The core challenge this paper addresses is developing a multi-view gaze estimation system that can generalize to arbitrary camera arrangements and unseen head poses without requiring extensive re-training or specialized datasets for each new setup.

Methods

The authors propose a "Rotation-Constrained Cross-View Feature Fusion" method. The core idea is to use the relative rotation matrix between two cameras as a constraint to guide feature fusion. Instead of simply concatenating features from different views, their network learns to extract "rotatable" features.

Here's a breakdown of their approach:

  • Input: The system takes two synchronized images (a target and a reference) and the relative rotation matrix ($R$) between the cameras that captured them.
  • Feature Extraction: Standard backbone networks (like ResNet-50) extract initial features from each image.
  • Rotatable Feature Extractor: A dedicated module takes these backbone features and transforms them into "rotatable" 3D vectors.
  • Rotation-Constrained Feature Fusion Blocks: These are the heart of the method. In each block, the rotatable features from one view are explicitly rotated using the matrix $R$ and then fused with the backbone features of the other view. This process is repeated across multiple stacked blocks. The network is trained to learn features that are amenable to this rotation operation, effectively making them "pose-invariant" in a relative sense.
  • Gaze Estimation: Finally, fused features are used by a gaze estimator to predict the 3D gaze direction.

The key novelty lies in using the rotation matrix not as an additional input feature, but as an explicit operation within the fusion process, forcing the network to learn features that can be meaningfully transformed between views.

Datasets

The paper uses two datasets for training and evaluation:

  • ETH-XGaze: A real-world dataset with 110 participants captured by 18 synchronized cameras. For generalization evaluation, they split participants into training and testing sets and also split cameras into training and testing sets.
  • MPII-NV: A synthetic dataset created by rendering images from the MPIIFaceGaze dataset with varying head poses and camera positions. This dataset allows for precise control over camera extrinsics and head poses.

They specifically highlight the challenge of generalizing to unseen head poses and camera configurations, which their experimental setup aims to simulate.

Key Insights and Ideas

  • Rotation as a Constraint, Not Just Input: The most significant insight is treating the relative camera rotation as an active constraint in the feature fusion process. This forces the network to learn features that are inherently adaptable to different viewpoints, rather than just trying to learn a universal representation.
  • Adaptive Feature Fusion: By iteratively fusing rotated features with backbone features, the model can adaptively leverage information from the reference view to improve the target view's representation, especially for challenging head poses.
  • Generalization Beyond Domain Adaptation: The method achieves generalization without requiring target domain data, distinguishing it from domain adaptation techniques. This makes it more practical for real-world deployment where new environments or camera setups might be encountered.
  • Head Pose is the Primary Generalization Hurdle: The paper strongly emphasizes that unseen head poses pose a greater generalization challenge than factors like illumination or individual identity, justifying their focus on this aspect.

Key Results

  • Superior Generalization: The proposed method significantly outperforms baseline methods (including single-image and other multi-view approaches) on unseen head poses, showing improvements of up to 29% in angular error on the ETH-XGaze dataset.
  • Competitive Performance: Even in seen head pose conditions, the method achieves competitive or superior results compared to baselines.
  • Cross-Dataset Performance: The model demonstrates strong generalization across different datasets, indicating its robustness to variations in data distribution.
  • Minor Inference Overhead: While multi-view methods inherently have higher computational costs, the proposed approach adds only a minor inference time increase (roughly doubling that of a single-image baseline) compared to its significant performance gains.

Limitations and Future Work

  • Hardware Requirements: The primary limitation acknowledged is the slightly increased hardware requirement due to the multi-view nature of the approach, though it's noted as being comparable to existing eye trackers.
  • Sensitivity to Rotation Matrix Accuracy: The paper notes that the model can be sensitive to errors in the rotation matrix at inference time, especially when using estimated head poses rather than calibrated camera extrinsics.
  • Future Work: The authors suggest developing lightweight models that are more robust to errors in rotation matrix estimation and time synchronization. They also hint at exploring self-learning transformations for improved gaze and head redirection.

Summary updated

BibTeX
@inproceedings{hisadome2024rotation,
  author = {Yoichiro Hisadome and Tianyi Wu and Jiawei Qin and Yusuke Sugano},
  title = {{Rotation-Constrained Cross-View Feature Fusion for Multi-View Appearance-Based Gaze Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}

Loading related content...

Figure