Will you be Aware? Eye Tracking-Based Modeling of Situational Awareness in Augmented Reality

Zhehan Qu, Tianyi Hu, Christian Fronk, Maria Gorlatova

Introduction

Augmented Reality (AR) offers transformative potential for high-stakes environments, such as emergency medicine and industrial maintenance. However, the integration of virtual overlays into the physical field of view introduces a "salience paradox": the very tools designed to enhance performance can inadvertently monopolize attentional resources. This phenomenon, known as "cognitive tunneling," occurs when a user becomes so focused on virtual cues that they miss critical real-world events.

This paper investigates the feasibility of predicting a user's Situational Awareness (SA) in real-time using eye-tracking data and a novel Graph Neural Network (GNN) architecture, FixGraphPool. By analyzing how users distribute their attention during a simulated medical emergency, the researchers demonstrate that cognitive states can be objectively modeled using sensors standard in modern AR headsets.

The Challenge of Situational Awareness in AR

Situational Awareness—the perception and comprehension of environmental elements and the projection of their status—is critical in safety-sensitive tasks. In an AR-assisted CPR scenario, a responder might become so fixated on a virtual compression gauge that they fail to notice a patient's deteriorating condition, such as vomiting or bleeding.

Traditional methods for measuring SA, such as the Situation Awareness Global Assessment Technique (SAGAT), are intrusive and retrospective, requiring the task to be paused for questionnaires. This research seeks an objective, sensor-based alternative that can operate continuously without interrupting the user's workflow.

Methodology: FixGraphPool

The researchers developed FixGraphPool, a GNN designed to model the complex spatial and temporal dynamics of human gaze in unconstrained 3D environments. Unlike traditional models that treat eye-tracking data as simple linear sequences of coordinates, FixGraphPool constructs a graph representation of visual attention:

  • Nodes: Represent individual fixations, capturing duration, location, and the specific target (e.g., an AR widget versus the physical mannequin).
  • Edges: Represent two types of connections. Temporal edges follow the chronological sequence of looks, while Spatial edges connect gaze points that are physically close to one another on the task object.
  • Edge Pooling: The model employs a pooling mechanism that iteratively contracts edges, simplifying the graph to identify high-level patterns in how a user shifts their focus between the virtual and real worlds.

Experimental Design

A study was conducted with 36 participants using a Magic Leap 2 headset. Participants performed CPR on a mannequin guided by an AR interface providing real-time visual and auditory feedback. To test SA, researchers introduced "unexpected incidents"—simulated vomiting and bleeding—that required the participant to deviate from the primary CPR task.

Data was collected at 60Hz, capturing gaze direction, pupil dilation, and fixation targets. SA was labeled as "good" or "poor" based on a combination of observer records (did the user react to the incident?) and freeze-probe questionnaires assessing the user's perception and decision-making.

Key Findings and Insights

The study revealed several critical insights into the visual signatures of situational awareness:

  • The "Scanning" Signature: Participants with high SA exhibited significantly larger saccadic amplitude and velocity. In practice, this means aware individuals move their eyes faster and across wider distances, actively scanning the environment rather than staring intently at AR widgets.
  • Virtual Over-Fixation: Poor SA was strongly correlated with a higher proportion of time spent fixating on virtual content. This confirms the cognitive tunneling hypothesis: the AR UI itself can become a primary distraction.
  • Predictive Performance: FixGraphPool achieved an accuracy of 83.0% (81.0% F1-score) in predicting SA levels within a 7-second window. This significantly outperformed traditional machine learning models (e.g., Logistic Regression at 76.4%) and state-of-the-art time-series models (e.g., PatchTSMixer at 74.5%), which struggled with the non-linear nature of gaze behavior.
  • The Multitasking Irony: Interestingly, participants who self-reported a high ability to shift attention actually showed less dynamic visual processing and lower SA during the trials, suggesting a gap between perceived and actual multitasking performance under stress.

Conclusion and Future Work

This work demonstrates that eye-tracking data, when processed through a spatial-temporal graph model, can reliably predict cognitive states in AR. While the current findings are tied to the specific UI layout used in the study, they provide a foundation for "closed-loop" AR systems.

Such systems could detect declining SA in real-time and automatically adjust the interface—for instance, by dimming virtual elements or simplifying the display—to "force" the user's attention back to the physical environment. Future research will focus on generalizing these models across different interface designs and hardware constraints to ensure safety in diverse AR applications.

Summary updated

BibTeX
@inproceedings{qu2025will,
  author = {Zhehan Qu and Tianyi Hu and Christian Fronk and Maria Gorlatova},
  title = {{Will you be Aware? Eye Tracking-Based Modeling of Situational Awareness in Augmented Reality}},
  booktitle = {ISMAR},
  year = {2025},
  doi = {10.1109/ismar67309.2025.00051},
}
Figure