Standard deep learning models often struggle with "spurious correlations"—they might classify a sentence as sarcastic based on a specific keyword rather than understanding the underlying linguistic nuance. Humans, however, process information through a selective attention mechanism: our eyes linger on complex words, skip over predictable ones, and dart back to resolve ambiguities.
A growing body of research suggests that we can bridge this gap by using human eye-tracking data—often called "gaze behavior"—as a supervision signal. By teaching models to predict where a human would look while performing a task, we instill a "cognitive inductive bias" that aligns machine attention with human reasoning.
Gaze as an Auxiliary Supervision Signal
The most common way to integrate gaze is through Multi-Task Learning (MTL). Instead of training a model solely on a primary task (like sentiment analysis), researchers like Abhijit Mishra and Pushpak Bhattacharyya have shown that adding an auxiliary task—predicting gaze metrics like fixation duration or saccades—forces the model to learn more robust internal representations.
In Cognition-Cognizant Sentiment Analysis With Multitask Subjectivity Summarization, the authors demonstrate that a model trained to predict human gaze becomes significantly better at "subjectivity extraction." The intuition is simple: if a human reader fixates on a specific adjective to determine sentiment, the model should learn that this adjective is more "cognitively significant" than a neutral noun. This shared feature extraction backbone ensures that the model's internal saliency maps mirror human-like selective attention.
The Synergy of Gaze and Action in Vision
In computer vision, particularly egocentric (first-person) video, gaze is more than just a signal—it is a functional index of intent. When you make a cup of tea, your eyes precede your hands, identifying the kettle before you reach for it.
Yin Li and colleagues argue in In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video that gaze and action should be modeled jointly rather than independently. By treating gaze as a probabilistic latent variable, their framework allows action recognition to guide gaze estimation and vice versa. This "mutual context," as explored by Yifei Huang in the Mutual Context Network, creates a feedback loop: knowing the action (e.g., "pouring") narrows down where the gaze should be (the cup), while the gaze location provides the spatial context needed to identify the action accurately.
Dynamic Computation and Cognitive Load
Beyond just "where" a model looks, gaze data tells us "how hard" a human is thinking. Longer fixations typically correlate with higher cognitive load or processing difficulty.
Nora Hollenstein and Xinting Huang took this literally in Longer Fixations, More Computation: Gaze-Guided Recurrent Neural Networks. They designed an architecture where predicted gaze duration controls the activation of neural layers. If the "gaze" prediction suggests a word is difficult, the model allocates more parallel RNN components to process it. This mimics the biological reality that we don't spend equal effort on every token; we vary our computational budget based on the complexity of the input.
Generalization and the Free Lunch
One of the most compelling reasons to use gaze is its ability to improve zero-shot generalization. In Cognitively Aided Zero-Shot Automatic Essay Grading, Sandeep Mathias and Diptesh Kanojia found that models trained with gaze as an auxiliary task performed 5% better on entirely new essay topics. Because human reading patterns are relatively consistent across topics, gaze acts as a topic-independent proxy for text quality.
Furthermore, this data is often a "neglected free lunch." As Dongyoon Han points out in Neglected Free Lunch - Learning Image Classifiers Using Annotation Byproducts, we already collect behavioral data (like mouse traces and hover times) during the standard annotation process. By simply not throwing this data away and using it as an auxiliary supervision signal, we can improve model robustness and foreground-background separation at zero additional cost.
Why It Works: Alignment as Regularization
The success of these methods suggests that human gaze acts as a powerful regularizer. By forcing a model's attention weights to align with human fixations, we prevent the model from over-fitting to noise or "cheating" by using background pixels or irrelevant words. Whether it is Jonathan Malmaud using gaze to improve Machine Reading Comprehension or Ritwik Bera using it for Imitation Learning, the core principle remains the same: human behavior provides a roadmap for task-relevant information that raw data alone often obscures.