Evaluating Cross-Subject and Cross-Device Consistency in Visual Fixation Prediction

Yuli Wu, Henning Konermann, Emil Mededovic, Peter Walter, Johannes Stegmaier

Abstract

As eye-tracking technology transitions from controlled laboratory environments to real-world applications, validating the reliability of consumer-grade wearables is essential. This study investigates the consistency of visual fixation data between the Meta Aria Glasses and high-end stationary systems using the MIT1003 benchmark. Our findings indicate that while individual gaze variability remains high, group-level saliency maps are remarkably consistent across devices. These results suggest that wearable eye trackers are sufficiently accurate for applications in neuroprosthetics and visual saliency modeling.

Introduction

The shift toward mobile eye tracking presents a significant opportunity for fields ranging from consumer behavior to medical rehabilitation. However, it also introduces a critical technical question: can data from a consumer-grade wearable device, costing a fraction of professional equipment, provide a reliable "gold standard" for human gaze?

This question is particularly vital for the development of visual neuroprosthetics. For individuals with profound visual impairments, a prosthesis must intelligently prioritize which environmental information to stimulate in the brain. To achieve this, researchers rely on models of natural human gaze. This study evaluates whether the "average" gaze patterns captured by wearable devices are consistent enough to serve as the foundation for these life-altering technologies.

Methodology

To assess cross-device consistency, the researchers compared two distinct recording environments:

  1. Laboratory Baseline: Data from the MIT1003 dataset, recorded with a high-speed ISCAN ETL-400 stationary tracker (240Hz).
  2. Wearable Testbed: New data collected from 9 participants using Meta Aria Glasses, which utilize monochrome eye-tracking cameras (30Hz to 90Hz) and an RGB scene camera.

Data Processing Pipeline

Because the Aria Glasses record from an egocentric perspective, the team employed a ResNet-18 deep learning architecture to predict gaze coordinates from the eye-camera videos. These coordinates were then mapped onto the original static images from the MIT1003 dataset using homography (perspective transformations). This allowed for a direct, pixel-to-pixel comparison between the wearable data and the laboratory ground truth. Consistency was quantified using standard saliency metrics, including AUC-Judd, Normalized Scanpath Saliency (NSS), and Similarity (SIM).

Key Findings

1. Robustness of Group Saliency

The study found that group-level saliency maps (heatmaps representing the aggregate gaze of multiple subjects) are highly stable across devices. In simple scenes featuring a single, prominent focal point—such as a lone animal or a distinct object—gaze patterns were nearly identical regardless of the hardware used. This suggests that for many research applications, the sheer volume of participants can compensate for the lower sampling rates of wearable hardware.

2. The Individual-to-Average Gap

While group averages were robust, the researchers noted a persistent challenge in predicting individual behavior. The "individual-to-average" consistency remained weak, particularly as scene complexity increased. In crowded or visually dense environments, individual subjects exhibited unique scanning behaviors, making it difficult to predict a specific person's gaze at any given moment despite the stability of the group's aggregate focus.

3. Quantitative Performance

The Meta Aria Glasses demonstrated impressive performance relative to the laboratory baseline:

  • AUC-Judd: 0.837 (outperforming the single-observer baseline of 0.80).
  • NSS: 1.666.
  • Correlation Coefficient (CC): 0.496.

These metrics indicate a strong linear relationship between the heatmaps generated by the two classes of devices, proving that wearable data is highly representative of established laboratory standards.

Discussion and Future Directions

The success of the Aria Glasses in replicating laboratory-grade saliency maps opens new doors for "in-the-wild" research. However, the 30Hz sampling rate remains a limitation for capturing fine-grained temporal dynamics like micro-saccades.

Future work will focus on two primary areas:

  • Scene Simplification: To assist neuroprosthetic users, researchers are exploring techniques to simplify complex environments, helping users focus on the most salient information.
  • Foundation Models: The team intends to leverage models like DINOv2 to explore "zero-shot" gaze prediction. This could allow systems to predict where a human would look without requiring real-time eye-tracking data for every new environment.

Conclusion

This study confirms that consumer-grade wearable eye trackers can produce gaze data consistent with high-end laboratory systems at the group level. By releasing their new dataset of Aria-based fixations on MIT1003 stimuli, the authors provide a valuable resource for the community to further bridge the gap between controlled lab studies and real-world visual behavior.

Summary updated

Abstract

Understanding cross-subject and cross-device consistency in visual fixation prediction is essential for advancing eye-tracking applications, including visual attention modeling and neuroprosthetics. This study evaluates fixation consistency using an embedded eye tracker integrated into regular-sized glasses, comparing its performance with high-end standalone eye-tracking systems. Nine participants viewed 300 images from the MIT1003 dataset in subjective experiments, allowing us to analyze cross-device and cross-subject variations in fixation patterns with various evaluation metrics. Our findings indicate that average visual fixations can be reliably transferred across devices for relatively simple stimuli. However, individual-to-average consistency remains weak, highlighting the challenges of predicting individual fixations across devices. These results provide an empirical foundation for leveraging predicted average visual fixation data to enhance neuroprosthetic applications.
BibTeX
@misc{wu2025evaluating,
  author = {Yuli Wu and Henning Konermann and Emil Mededovic and Peter Walter and Johannes Stegmaier},
  title = {{Evaluating Cross-Subject and Cross-Device Consistency in Visual Fixation Prediction}},
  year = {2025},
  eprint = {2502.05554},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.05554},
}
Figure