Perturbation Profiles of Scanpath Classifiers: What Reading, Code-Comprehension and Video Decoders Are Sensitive To

Saleh Mozaffari, Abolfazl Nadi

When a machine learning model successfully predicts a person's cognitive state or task from their eye movements, we often celebrate the high accuracy and move on. But what is the model actually "seeing"? Is it capturing deep insights into reading behavior, or is it just noticing that one group of participants looked more at the left side of the screen?

In their paper, "Perturbation Profiles of Scanpath Classifiers," Saleh Mozaffari and Abolfazl Nadi introduce a systematic framework to answer these questions. By intentionally "breaking" specific properties of eye-tracking data and seeing how models react, they expose hidden confounds and clarify exactly what our gaze-based decoders are sensitive to.

TL;DR

The authors propose perturbation profiles: a method to audit scanpath classifiers by systematically destroying properties like fixation order or spatial structure and refitting the models. This approach successfully identifies stimulus confounds (e.g., models learning screen layout instead of "comprehension") and reveals the specific spatial and temporal scales at which different gaze descriptors operate.

What Problem Does This Paper Address?

Scanpath classification—predicting things like expertise, task, or comprehension from a sequence of fixations—is a powerful tool in human-computer interaction and psychology. However, these models are often "black boxes." A high Area Under the Curve (AUC) score doesn't tell you if the model relies on complex temporal sequences, simple spatial dispersion, or artifactual confounds in the experimental setup.

The authors identify a critical gap: standard performance metrics can't distinguish between a model that has learned a genuine cognitive process and one that has merely exploited a "shortcut" (like a difference in stimulus layout between two classes).

Methods

The core contribution is the perturbation profile. The researchers take a trial-level scanpath and apply five distinct transformations, each designed to destroy one specific property while keeping others invariant:

  1. Order: Permutes the sequence of fixations, destroying temporal information.
  2. Duration Coupling: Scrambles which fixation duration belongs to which location, breaking the link between "where" and "how long."
  3. Duration Marginal: Replaces durations with random draws from a global pool, destroying the trial's specific timing distribution.
  4. Rotation: Applies rigid rotation and reflection, removing orientation and "side" information while preserving internal geometry.
  5. Jitter: Adds Gaussian noise at increasing scales (from character-level to line-level), blurring spatial structure.
Visualizing the five scanpath transformations used to build a perturbation profile.

Crucially, the authors refit the classifier on the perturbed data. This measures the sensitivity of the entire pipeline. If the AUC drops significantly after a transformation, the pipeline was sensitive to that property. If it stays the same, the information was either unused or redundant.

Datasets

The authors validated their method on twelve binary classification problems across three public datasets:

  • SB-SAT: Reading SAT passages (95 participants). Tasks included distinguishing native vs. non-native readers and high vs. low comprehension.
  • EMIP: Programmers reading code (up to 159 participants). Tasks included distinguishing between different code snippets, expertise levels, and answer correctness.
  • GazeBase: A large-scale dataset of reading, video viewing, and game tasks (60 participants).

Key Insights and Ideas

  • The "Negative Sensitivity" Surprise: Counter-intuitively, destroying information (like fixation order) sometimes improves a classifier's performance. For example, permuting order can turn a "mean saccade amplitude" feature into a pure measure of "spatial spread," which might be a more powerful signal for certain tasks.
  • Exposing Confounds: In the EMIP dataset, a classifier initially appeared to decode "comprehension" with an AUC of 0.70. The perturbation profile revealed a rotation sensitivity, leading the authors to discover that most "wrong" answers occurred on one specific program with a unique layout. The model wasn't decoding comprehension; it was decoding which program was on the screen.
  • Scale Matters: By sweeping the "jitter" transformation, the authors could pinpoint the exact spatial scale of a behavior. For native vs. non-native readers, they found that recurrence-based models were sensitive to structure at the scale of text lines, but indifferent to jitter at the scale of individual characters.
  • Refitting is Essential: Simply re-scoring a pre-trained model on perturbed data is misleading. Refitting allows the learner to adapt to redundant information, providing a truer measure of what the pipeline (descriptor + learner) can actually achieve.

Key Results

  • Confound Detection: When the program layout confound in EMIP was removed, the "comprehension" AUC dropped from 0.70 to a near-chance 0.47, proving the original result was an artifact.
  • Descriptor Strengths: A novel Fixation Adjacency Graph (FAG) anchored to code tokens outperformed all other structured descriptors on program-related tasks (reaching 0.78 AUC, and 0.87 with boosted trees).
  • Reading Dynamics: Native and non-native readers were discriminated (AUC 0.65) primarily through the sequential structure of "returns" (rereading and regressions) at the line scale, rather than where eyes landed within a word.

Limitations and Future Work

  • Non-Orthogonality: The transformations aren't perfectly independent; for instance, jittering locations naturally changes saccade amplitudes and recurrence rates.
  • Binary Focus: The current framework is designed for binary classification. Extending it to regression (e.g., predicting a continuous test score) is a logical next step.
  • Pipeline Specificity: A sensitivity is a property of the entire pipeline, including the specific learner (e.g., SVM vs. XGBoost). The authors found that switching learners can occasionally change which properties a model relies on.
  • Interpretative, Not Causal: The profile tells us what the model uses, not necessarily what the human participant was doing cognitively.

The authors have released their method as an open-source Python package, scanpathdecomp, encouraging the community to make perturbation profiling a routine part of scanpath analysis.

Summary updated

Abstract

When a scanpath descriptor discriminates a task, it is rarely clear what the discrimination rests on. We propose a perturbation profile for trial-level descriptors. Five transformations each destroy one stated property while preserving a listed set of invariants: fixation order, duration-position coupling, the duration distribution, orientation and handedness, and fine spatial structure. The classifier is refit, and the drop in participant-held-out AUC is the pipeline's sensitivity to that property. Because refitting lets the learner use redundant information, this describes the refitted pipeline, not what a fitted classifier used. Intervals come from a participant bootstrap that refits; in simulations at the sample sizes analysed, where the target was zero they excluded zero at a pooled rate of at most 2.5%, and they covered the target in 99.6% of 2720 cases. We profiled seven descriptors on twelve binary problems from three public datasets: reading (SB-SAT, 95 participants), program comprehension (EMIP, up to 159) and video and game viewing (GazeBase, 60). Permuting fixation order raised the AUC of a descriptor built on mean saccade amplitude on three problems (by up to 0.23), because permutation turns amplitude into a measure of spatial spread. A "comprehension" contrast that pooled two programs reached AUC 0.70 with a spread-only baseline and fell to 0.47 within one program: the profile located the confound in stimulus layout. On native versus non-native readers, recurrence was sensitive to order and to jitter at the scale of text lines but not of characters, while a two-feature duration and amplitude descriptor reached the same AUC (0.65). A graph descriptor built on the code's tokens was the strongest non-baseline descriptor on the program contrast (0.78; 0.87 with boosted trees). The interpreted results held under longer runs and three seeds; under a second learner 15 of 16 kept t
BibTeX
@misc{mozaffari2026perturbation,
  author = {Saleh Mozaffari and Abolfazl Nadi},
  title = {{Perturbation Profiles of Scanpath Classifiers: What Reading, Code-Comprehension and Video Decoders Are Sensitive To}},
  year = {2026},
  eprint = {2611.11632},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11632},
}
Figure