When we read a sentence, inspect an X-ray, or navigate an augmented reality environment, our eyes do not move in a smooth, continuous stream. Instead, they perform a series of rapid jumps called saccades, punctuated by brief pauses called fixations where the brain processes visual information.
Traditionally, researchers analyzed this behavior using aggregate statistics, such as total fixation duration or overall saccade counts. However, these metrics discard a vital dimension of human cognition: its non-linear, relational structure. If a reader encounters a confusing word or a sarcastic twist, they do not simply look at it longer; they look back at previous words to re-evaluate context.
To capture these intricate behavioral patterns, a powerful paradigm has emerged: transforming raw eye-tracking data into scanpath networks. By modeling gaze as a graph, researchers can use network science and deep learning to decode a user's comprehension, expertise, and situational awareness.
The Anatomy of a Gaze Graph
The core intuition behind this approach is that cognitive processing is inherently structural. To map eye movements to a graph, we translate the spatial and temporal components of a scanpath into nodes and edges:
- Nodes (What we look at): Depending on the task, nodes can represent words in a text, predefined Areas of Interest (AOIs) on an image, or 3D spatial coordinates in an environment. Crucially, if a user revisits the same word or area multiple times, those visits are merged into a single unique node.
- Edges (How we move between them): Directed edges represent saccades—the transitions from one fixation point to the next. These edges can be weighted by the frequency of transitions or by the duration of the fixations they connect.
Linear Reading (Low Cognitive Load):
[Word 1] ──> [Word 2] ──> [Word 3] ──> [Word 4]
Non-Linear Reading (High Cognitive Load / Regressions):
[Word 1] <───> [Word 2] ──> [Word 3]
│ ▲
└───────────────────────────┘
This transformation shifts the analytical focus from how long someone looks at something to how they connect different pieces of information. A simple linear progression yields a chain-like graph, whereas a complex, recursive thought process generates a dense network rich with loops and central hubs.
Quantifying Cognitive Complexity in Text
One of the earliest and most successful applications of gaze graphs is in Natural Language Processing (NLP), particularly for tasks where the literal meaning of words conflicts with the intended sentiment, such as sarcasm.
In their work on sarcasm detection, Abhijit Mishra et al. demonstrated that reading patterns form highly non-linear graphs when a reader encounters ironic or sarcastic text. When the literal words do not match the context, the reader's eyes backtrack, creating a high density of regression edges. By calculating graph-theoretic metrics like edge density and weighted degree, the authors quantified this cognitive friction, allowing machine learning models to detect sarcasm more effectively than by looking at text alone.
The authors extended this insight to broader sentiment analysis in Leveraging Cognitive Features for Sentiment Analysis. They modeled eye movements as "gaze-saliency graphs," proving that structural properties like graph density capture the "surprisal factor" of complex sentences.
This topological signature changes not just with text difficulty, but also with familiarity. For instance, Yoav Meiri et al. investigated whether eye movements could reveal if a person is reading a text for the first time or the second. They discovered that text familiarity fundamentally alters the network's topology, shifting the "hubs" of attention and lowering the overall density of the saccade network as the reading path becomes more efficient.
Similarly, Xiaochuan Ma et al. used scanpath networks to study individual differences in reading comprehension. They found that skilled readers exhibit highly optimized network properties: higher global efficiency and lower transitivity, indicating fewer redundant regressions and more effective word skipping.
Mapping Expertise and Awareness in Visual Domains
The utility of gaze graphs extends far beyond the left-to-right constraints of reading text. In two-dimensional and three-dimensional spaces, graph topology becomes an even more critical tool for understanding visual strategy.
In medical imaging, diagnosing an illness requires a highly systematic search pattern. Pingping Yang et al. constructed ego networks from dental students' eye-tracking data as they learned to interpret X-rays. By analyzing eleven network metrics—including centrality measures like PageRank—the authors showed that "expert-like" performance correlates with highly interconnected networks. Experts do not just look at anomalies; they rapidly transition between the anomaly and healthy reference structures, creating a well-integrated network topology that reflects superior diagnostic reasoning.
Novice Scanpath Graph (Unstructured, fragmented):
[AOI A] ──> [AOI B] [AOI C] ──> [AOI D]
Expert Scanpath Graph (Integrated, high centrality):
[AOI A] <───> [Diagnostic Hub] <───> [AOI C]
▲
│
[AOI B]
When moving to dynamic environments like Augmented Reality (AR), the temporal sequence and 3D spatial proximity must be modeled simultaneously. In Will You Be Aware? Eye Tracking-Based Modeling of Situational Awareness in Augmented Reality, Zhehan Qu et al. tackled the problem of "cognitive tunneling"—where users focus so heavily on digital overlays that they miss critical real-world events.
The authors developed a Graph Neural Network architecture called FixGraphPool. Instead of treating gaze as a simple chronological timeline, their model constructs a graph where nodes encode fixation durations and edges capture spatial-temporal transitions. They demonstrated that modeling these spatial relationships as a network is significantly more predictive of a user's internal cognitive state and situational awareness than analyzing chronological sequences alone.
Deep Learning and Attention Sparsification
As gaze graphs grow in complexity, researchers are moving beyond manual feature engineering (like calculating average node degrees) toward end-to-end deep learning architectures.
In the domain of computer-aided diagnosis, Naji Khosravan et al. introduced a collaborative system (C-CAD) that embeds a radiologist's gaze graph directly into a deep learning pipeline. Because human gaze can contain noise or irrelevant fixations, the authors utilized an attention-based sparsification model. This mechanism automatically prunes weak or accidental edges in the gaze graph, isolating the core structural backbone of the expert's visual search. The resulting sparse graph acts as an attentional prior, guiding convolutional neural networks to focus on the most clinically relevant regions of an image.
Why the Idea Works
Scanpath network analysis works because it bridges the gap between raw behavioral data and high-level cognitive architectures. Human thoughts are not scalar values, nor are they strictly linear sequences; they are webs of associations, comparisons, and revisions.
By projecting eye movements into a graph structure, this research direction ensures that the mathematical representation of the data matches the underlying topology of human cognition. Whether it is a Graph Neural Network predicting a pilot's awareness in AR, or a centrality metric indexing a student's reading comprehension, the graph abstraction successfully distills the chaotic, rapid movements of our eyes into a structured map of the human mind.
Key Foundational Work
Leveraging Cognitive Features for Sentiment Analysis (2016) — contributed 2 concept elements to this direction.
Harnessing Cognitive Features for Sarcasm Detection (2016) — contributed 1 concept elements to this direction.
A collaborative computer aided diagnosis (C-CAD) system with eye-tracking, sparse attentional model, and deep learning (2018) — contributed 1 concept elements to this direction.