Aligning Neural Network Attention with Human Gaze

133 concepts86 papers1 proposal

When we build "attention" mechanisms into modern deep learning architectures, we often assume the model learns to focus on the same features a human would. However, as Abhishek Das et al. demonstrated in Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions?, unsupervised attention weights optimized purely end-to-end are frequently fortuitous rather than intentional. Left to their own devices, neural networks routinely exploit dataset shortcuts, focusing on irrelevant background noise or spurious statistical correlations to minimize training loss.

To bridge this cognitive gap, a growing body of research focuses on explicit attention supervision. By treating human eye-tracking data, expert saliency maps, or cognitive reading signals as a direct training signal, researchers can force a model's internal attention layers to align with human visual and cognitive priorities.


The Core Mechanism: How Attention Alignment Works

In a standard neural network, attention weights are shaped implicitly through backpropagation from a primary task loss (such as cross-entropy). Explicit attention supervision modifies this objective by introducing an auxiliary loss function or regularization term that directly penalizes deviations from human gaze distributions.

Input Data ───► [ Internal Attention Layers ] ───► Predicted Attention Map
                         │                                 │
                         ▼                                 ▼
                 [ Human Gaze Prior ] ────────────► [ Auxiliary Loss ]
                 (Eye-tracking/Saliency)             (KL-Div / MSE / Hungarian)

Mathematically, this is achieved by minimizing the divergence between the model's internal attention maps and ground-truth human gaze patterns during training. For example, Chenxi Liu et al. in Attention Correctness in Neural Image Captioning incorporated an explicit attention loss term to ensure that image captioning models look at the correct spatial regions during token generation.

The alignment strategy can vary depending on the architecture:

  • Vision-Language Models: The authors of Gaze-VLM: Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding apply this as a training-time regularization signal. This allows the model to inherit human-centric visual prioritization during optimization, meaning it can be deployed using standard RGB video inputs without requiring eye-tracking hardware at inference time.
  • Set-Matching Objectives: In tasks like zero-shot learning, Yang Liu et al. in Goal-Oriented Gaze Estimation for Zero-Shot Learning treated predicted attention maps and human gaze points as a set-matching problem, optimizing the alignment via the Hungarian algorithm to enforce biologically plausible search strategies.
  • Multi-Task Regularization: In video understanding, Petros Koutras demonstrated in SUSiNet: See, Understand and Summarize It that explicitly supervising spatio-temporal attention maps with eye-tracking data acts as a powerful regularizer, boosting downstream performance across multiple tasks like action recognition and video summarization.

Why It Works: Mitigating Shortcut Learning and Causal Confusion

Forcing a model to "look" like a human works because it introduces a highly effective inductive bias that eliminates non-causal features. Deep networks are notorious for picking up on environmental artifacts—a phenomenon known in imitation learning as causal confusion, where an agent misattributes the cause of an action to a distracting background variable.

By penalizing discrepancies between model focus and expert gaze, frameworks like GABRIL, developed by Amin Banayeeanzade et al. in Gaze-Based Regularization for Mitigating Causal Confusion in Imitation Learning, force agents to isolate task-relevant features (such as the road or a game objective) from environmental noise.

This principle applies equally to text. In natural language processing, Large Language Models (LLMs) frequently rely on arbitrary word co-occurrences rather than true semantic reasoning. Yuqi Ren introduced HuaSLIM in HuaSLIM: Human Attention Motivated Shortcut Learning Identification and Mitigation for Large Language models, demonstrating that regularizing self-attention maps with human reading gaze prevents LLMs from relying on lexical overlaps.

Ultimately, grounding the network's internal importance-weighting in human-validated reasoning patterns—as seen in the HINT framework by Ramprasaath R. Selvaraju et al. (Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded)—ensures that the model remains robust against adversarial shifts and out-of-distribution data.


Transferring Implicit Domain Expertise

In specialized domains like medical imaging, explicit attention supervision serves a dual purpose: it improves performance and acts as a vehicle for transferring implicit expert knowledge.

As noted by Jianwei Niu et al. in A survey on incorporating domain knowledge into deep learning for medical image analysis, integrating radiologist eye-tracking data provides critical spatial hints that help models overcome the chronic shortage of labeled medical data. For instance, Yi Pan et al. in EG-SpikeFormer: Eye-Gaze Guided Transformer on Spiking Neural Networks for Medical Image Analysis used an attention alignment loss to ground a Spiking Neural Network's decisions in professional diagnostic logic, successfully mitigating shortcut learning in complex clinical classifications.

Crucially, this transfer depends strictly on the expertise of the human observer. Yifei Huang 0002 et al. in Leveraging Human Selective Attention for Medical Image Analysis with Limited Training Data discovered that while expert gaze patterns significantly boost diagnostic accuracy, incorporating gaze data from non-experts can actually degrade model performance. This confirms that the network is not merely benefiting from generic visual saliency, but is actively absorbing specialized diagnostic reasoning.

Furthermore, this alignment yields features that are inherently more interpretable for human-AI collaboration. Aidan Boyd demonstrated in The Value of AI Guidance in Human Examination of Synthetically-Generated Faces that models trained with loss functions prioritizing human-identified regions serve as far more effective team collaborators, as their internal focus naturally mirrors human perception.


Scaling Beyond Physical Eye-Tracking

A major bottleneck for this paradigm is the high cost and complexity of collecting high-resolution gaze data. To scale human-centric attention supervision, researchers rely on two main strategies: large-scale open datasets and cognitive simulations.

For language modeling, massive corpora like the OneStop dataset by Yevgeni Berzak et al. (OneStop: A 360-Participant English Eye Tracking Dataset with Different Reading Regimes) provide hundreds of hours of reading gaze data, enabling the benchmarking and grounding of language models in human-centric processing constraints.

When real-time tracking hardware is entirely unavailable, simulated gaze models or interactive interfaces fill the gap:

  • Cognitive Simulations: In HuaSLIM, Yuqi Ren bypassed hardware limitations by using cognitive models to simulate human reading behavior and gaze duration, providing a scalable ground-truth attention prior.
  • Interactive Unblurring: In medical imaging, Lai Jiang et al. developed the LAG database in A Large-Scale Database and a CNN Model for Attention-Based Glaucoma Detection using a simulated eye-tracking method. Experts were required to progressively reveal parts of a blurred fundus image until a diagnosis could be made. This process successfully captured high-quality diagnostic attention maps, proving that human-aligned attention supervision can be scaled effectively even in data-constrained environments.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_human_centric_attention_supervision_0de2e319,
  title = {human-centric attention supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/human-centric-attention-supervision/rdr_0de2e319065e479a9101946a5372cdbf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_human_centric_attention_supervision_0de2e319_v1,
  title = {human-centric attention supervision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/human-centric-attention-supervision/rdr_0de2e319065e479a9101946a5372cdbf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake