Introduction
While Large Language Models (LLMs) demonstrate remarkable capabilities, their text processing mechanisms differ significantly from human cognitive processes. Humans read non-linearly, skipping familiar words, fixating on complex terms, and regressing to previous phrases to resolve ambiguities. These "scanpaths" provide a rich signal of language comprehension that standard linear processing ignores.
This research introduces a novel fine-tuning strategy that "grounds" pre-trained language models (LMs) in human cognitive processes using gaze supervision. By integrating a gaze-based auxiliary loss during training, the authors improve model performance on downstream tasks without increasing computational costs or data requirements at inference time.
Methodology: The Scanpath Branch
The core of the proposal is a dual-branch architecture utilized exclusively during the fine-tuning phase. This approach addresses the hurdles of data scarcity and deployment overhead by using synthetic gaze data as a "cognitive inductive bias."
The architecture consists of:
- The Standard Branch: A conventional fine-tuning path where a Transformer (e.g., BERT) processes text, and the
[CLS]token embedding is used for classification. - The Scanpath Branch: This branch generates synthetic scanpaths for input sentences using a pre-trained Eyettention model. Contextualized token embeddings from the Transformer are rearranged to match the sequence of human-like fixations. This reordered sequence is then processed by a Gated Recurrent Unit (GRU) "Scanpath Encoder."
The model is trained on a joint objective minimizing both the standard task loss and a weighted auxiliary scanpath loss. Crucially, the scanpath branch is discarded after training, ensuring the final model remains a standard Transformer compatible with existing deployment pipelines.
Experimental Setup
Evaluation was conducted across eight tasks in the General Language Understanding Evaluation (GLUE) benchmark. To test the effectiveness of gaze as a cognitive inductive bias, the authors focused on low-resource settings with limited training samples ($K = 200, 500, 1000$). The Eyettention model used for synthetic gaze generation was pre-trained on the CELER corpus, which contains eye-tracking data from native English speakers reading natural sentences.
Results and Analysis
The gaze-augmented models (+SP) consistently outperformed both standard fine-tuning and traditional text augmentation (EDA) across nearly all GLUE tasks, particularly in low-resource scenarios.
Key findings include:
- Performance Gains: BERT saw improvements of 2–3%, while RoBERTa improved by 1–2%. The smaller gain for RoBERTa suggests its robust pre-training already captures some of the structural nuances provided by gaze.
- Task Specificity: The method was most effective for tasks requiring deep structural understanding, such as CoLA (Linguistic Acceptability) and STS-B (Semantic Similarity).
- The Importance of Order: Ablation studies confirmed that the specific human-like sequence of fixations is vital; randomizing or shuffling the scanpath significantly degraded performance, proving that the cognitive order carries a unique signal.
Conclusion and Future Work
By treating gaze as a supervision signal rather than an input feature, this approach provides the benefits of multi-modal learning without the associated inference-time overhead. Gaze patterns act as a principled form of data augmentation, helping models learn faster from fewer examples by highlighting semantically and syntactically important tokens.
Future research may focus on improving the quality of synthetic scanpaths through advanced generators like diffusion models and extending the approach to specialized domains (e.g., medical or legal) where standard gaze patterns may not apply.
Summary updated