Abstract
Eye-tracking data, or "scanpaths," provide critical insights into human language processing but are difficult and expensive to collect. We present ScanDL (Scanpath Diffusion conditioned on Language), a novel discrete sequence-to-sequence diffusion model that generates realistic synthetic scanpaths. By refining entire sequences simultaneously, ScanDL captures complex reading behaviors like regressions more effectively than auto-regressive models. Our results show that ScanDL outperforms existing cognitive and machine learning baselines in similarity to human reading patterns and maintains psycholinguistic validity across multiple datasets.
Introduction: The Challenge of Human Reading Patterns
In psycholinguistics, scanpaths—the sequence of fixations and saccades made during reading—serve as a window into cognitive linguistic processing. This data is invaluable for grounding large language models (LLMs) in human-like behavior. However, the scarcity of eye-tracking data, which requires specialized hardware and lab environments, limits its application. Existing synthetic generation methods, such as the cognitive E-Z Reader model or LSTM-based architectures, often fail to capture the non-linear nature of human reading, particularly "regressions" where readers return to previous words for re-processing.
Methodology: Scanpath Diffusion
ScanDL addresses these limitations by framing scanpath generation as a diffusion process. Unlike auto-regressive models that predict fixations sequentially, ScanDL generates the entire sequence at once, allowing for better global context and more natural regressions.
The model utilizes several key components:
- Latent Diffusion: Stimulus text and scanpaths are mapped to a continuous embedding space. Gaussian noise is added to the scanpath sequence during the "noising" phase, while the linguistic context (stimulus text) remains uncorrupted.
- Joint Embedding: The model incorporates pre-trained BERT embeddings for semantic depth, alongside word indices and positional encodings.
- Transformer Denoising: A 12-block encoder-only Transformer serves as the core engine, learning to reconstruct clean scanpaths from noisy latent variables.
Experimental Setup
The model was trained and evaluated on two primary corpora:
- CELER L1: A large-scale dataset of native English speakers.
- ZuCo: A dataset used for "across-dataset" evaluation to test the model's generalization capabilities to new hardware and experimental conditions.
Key Results and Psycholinguistic Validity
Performance was measured using Normalized Levenshtein Distance (NLD), which quantifies the similarity between generated and actual human scanpaths. ScanDL achieved state-of-the-art results across various test scenarios, including "New Sentence," "New Reader," and "Across-Dataset" generalizations.
Beyond raw metrics, ScanDL demonstrates high psycholinguistic validity. Bayesian analysis confirms that the model mimics human sensitivities to:
- Word Length: Skipping shorter, more predictable words.
- Lexical Frequency: Fixating longer on rare words.
- Surprisal: Spending more time on unexpected linguistic tokens.
Notably, ScanDL's synthetic paths often resemble an "average human" more closely than individual human paths resemble each other, suggesting it captures the core statistical properties of reading behavior.
Discussion and Limitations
While ScanDL represents a significant advancement, several areas for future work remain. Currently, the model predicts fixation locations but not durations, which are equally vital in eye-tracking research. Additionally, the iterative nature of diffusion models makes inference more computationally expensive than traditional sequence models. Future iterations could also incorporate "reader-specific" conditionings to simulate diverse populations, such as non-native speakers or individuals with reading disorders.
Conclusion
ScanDL provides a robust framework for generating high-quality synthetic eye-tracking data. By bridging the gap between cognitive modeling and generative AI, it offers a scalable solution to the data scarcity problem in psycholinguistics and cognitively-grounded NLP.
Summary updated