Learning Grounded Reasoning from Human Gaze

Gabriel Sarch, Declan Campbell, Nicholas Budny, Kia Ghods, Katerina Fragkiadaki, Jonathan D. Cohen, Brenden M. Lake, Taylor W. WebbarXiv 2026 · arXiv

TL;DR

Researchers have developed GGR (Gaze-Guided Reasoner), a method that uses human eye-tracking data as a "spatial prior" to train Vision-Language Models (VLMs). By teaching models to look where humans look—without forcing them to mimic every noisy fixation—GGR achieves state-of-the-art task performance while significantly improving the reliability and human-alignment of its visual reasoning.

What Problem Does This Paper Address?

While modern VLMs can perform complex reasoning, they often suffer from a "grounding" problem: their internal textual reasoning steps (Chain-of-Thought) frequently disconnect from the actual visual evidence in the image. This leads to hallucinations, where a model claims to see something that isn't there, or "binding" errors, where it confuses attributes between different objects.

Human gaze provides a natural blueprint for how to solve these tasks. When we look at a chart or search for an object, our eyes move systematically to extract relevant evidence. However, human gaze is also "noisy"—it includes redundant fixations, distractions, and reflects biological constraints (like limited peripheral vision) that AI models don't share. The core challenge this paper addresses is: How can we effectively transfer the "wisdom" of human visual exploration to VLMs without inheriting the noise of human biology?

Methods

The authors propose a multi-stage pipeline to create the Gaze-Guided Reasoner (GGR):

  1. Data Collection: They recorded 7,109 trials of humans performing visual tasks (counting, searching, and reading charts) while wearing eye-trackers.
  2. Gaze-Guided Annotation: Instead of training a model directly on raw gaze coordinates, they used a powerful "teacher" model (Qwen3.5-122B). This teacher was given the image, the question, and the human gaze points. It was tasked with generating a reasoning trace where it selects the most relevant gaze points to explain its logic.
  3. Supervised Fine-Tuning (SFT): A smaller, more efficient VLM (Qwen3-VL-8B) was trained to mimic these high-quality, gaze-informed reasoning traces.
  4. Reinforcement Learning (RL): Finally, they applied RL (using the GSPO algorithm) to optimize for task accuracy. Crucially, the RL stage did not use gaze data, yet it further improved the model's alignment with human gaze by reinforcing the effective exploration strategies learned during SFT.
The GGR training pipeline: from human gaze collection to gaze-guided annotation and RL.

Datasets

The study utilized a mix of newly collected data and established benchmarks:

  • Human Gaze Dataset (New): 7,109 trials across 16 different source datasets, including Visual Probe, PixMo, TallyQA, and ChartQA.
  • Evaluation Benchmarks: The models were tested on a diverse suite including ChartQA and ChartQA-Pro (reasoning), CountBenchQA and CountQA (counting), and MME-RealWorld-Lite and Visual Probe (visual search).

Key Insights and Ideas

  • Prior over Sequence: The most critical finding is that gaze is most effective when used as a flexible spatial prior rather than a strict sequence to be imitated. Models forced to explain every human fixation (the "Gaze-All" condition) performed worse than models allowed to selectively use gaze points (GGR).
  • RL as an Alignment Tool: Surprisingly, optimizing for accuracy via RL actually increased the model's alignment with human gaze. This suggests that human-like exploration is fundamentally a high-reward strategy for solving complex visual problems.
  • Exploration vs. Verification: Behavioral analysis revealed that gaze-trained models (GGR) act more like humans: they spend more time exploring the image before committing to an answer. In contrast, standard models often jump to a conclusion quickly and spend their "thinking" time verifying that conclusion post-hoc.
Qualitative comparison showing GGR performing useful exploration compared to baselines that either underexplore or over-verify.

Key Results

  • Superior Alignment: GGR-RL achieved a gaze alignment score of 1.15 bits/fix, significantly higher than the "No-Gaze-RL" baseline (0.99 bits/fix).
  • Strong Performance: GGR-RL matched or exceeded the accuracy of the strongest baselines across all tasks, achieving an overall accuracy of 68.8% on the evaluation suite.
  • Reliable Grounding: Human evaluators and GPT-4o-based metrics confirmed that GGR's reasoning claims were more likely to be supported by the image regions it pointed to. Specifically, GGR-RL produced "supported claims + points on/near target" in 74.9% of cases, compared to 70.5% for the no-gaze version.
Figure 1. GGR achieves higher human gaze alignment and better visual grounding while maintaining strong task performance. (Left) Mean accuracy versus human fixation log-likelihood relative to uniform (± SEM); open/filled circles denote pre/post-RL models. (Middle) Human gaze, Base RL, and GGR grounding traces on a held-out example. (Right) Supported grounded reasoning claims with points on/near target for GGR-RL and No-Gaze-RL (± SEM). All results on Qwen3-VL-8B-Instruct.

Limitations and Future Work

  • Architecture Specificity: The study primarily focused on the Qwen-VL architecture. It remains to be seen if these gaze-transfer benefits hold equally across different VLM families (e.g., Flamingo-style vs. LLaVA-style).
  • Biological Divergence: AI models do not have a fovea (the high-resolution center of the human eye) or the same working memory constraints as humans. The authors suggest that future work could explore adding these "biologically-plausible" constraints to VLMs to see if it further improves reasoning.
  • Scaling: While 7,000 trials is a significant dataset, it is small compared to the massive web-scale datasets used for general VLM training. Scaling gaze-data collection remains a logistical challenge.

Summary updated

Abstract

Human visual reasoning is driven by systematic exploration and extraction of task-relevant visual evidence. Human gaze provides a window into this process, making gaze a useful source of guidance for visually-grounded reasoning in vision-language models (VLMs). However, human gaze also includes noise and redundancies that may transfer poorly to VLMs. In this work, we investigate how human gaze data can be effectively leveraged to improve visual reasoning in VLMs, while maintaining alignment with human behavior. We collect 7,109 human gaze trials from a diverse and challenging set of visual reasoning tasks, and train grounded reasoning models with different forms of gaze guidance. Across evaluations in chart understanding, visual search, and counting, we find that using human gaze as a prior for grounded reasoning consistently outperforms supervision using the full gaze sequence. Based on these findings, we introduce GGR (Gaze-Guided Reasoner), which achieves greater human alignment and more reliable visual grounding while matching or outperforming leading grounded-reasoning baselines in accuracy. Reinforcement learning further improves task accuracy, gaze alignment, and grounding quality despite using no gaze supervision. Behavioral analyses show that gaze supervised models exhibit more visual exploration before arriving at an answer. Our results suggest that the value of human gaze lies in guiding spatial exploration and shaping reasoning behavior without requiring complete imitation of human behavior.
BibTeX
@misc{sarch2026learning,
  author = {Gabriel Sarch and Declan Campbell and Nicholas Budny and Kia Ghods and Katerina Fragkiadaki and Jonathan D. Cohen and Brenden M. Lake and Taylor W. Webb},
  title = {{Learning Grounded Reasoning from Human Gaze}},
  year = {2026},
  eprint = {2611.23209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23209},
}
Figure