TL;DR
Researchers have developed GGR (Gaze-Guided Reasoner), a method that uses human eye-tracking data as a "spatial prior" to train Vision-Language Models (VLMs). By teaching models to look where humans look—without forcing them to mimic every noisy fixation—GGR achieves state-of-the-art task performance while significantly improving the reliability and human-alignment of its visual reasoning.
What Problem Does This Paper Address?
While modern VLMs can perform complex reasoning, they often suffer from a "grounding" problem: their internal textual reasoning steps (Chain-of-Thought) frequently disconnect from the actual visual evidence in the image. This leads to hallucinations, where a model claims to see something that isn't there, or "binding" errors, where it confuses attributes between different objects.
Human gaze provides a natural blueprint for how to solve these tasks. When we look at a chart or search for an object, our eyes move systematically to extract relevant evidence. However, human gaze is also "noisy"—it includes redundant fixations, distractions, and reflects biological constraints (like limited peripheral vision) that AI models don't share. The core challenge this paper addresses is: How can we effectively transfer the "wisdom" of human visual exploration to VLMs without inheriting the noise of human biology?
Methods
The authors propose a multi-stage pipeline to create the Gaze-Guided Reasoner (GGR):
- Data Collection: They recorded 7,109 trials of humans performing visual tasks (counting, searching, and reading charts) while wearing eye-trackers.
- Gaze-Guided Annotation: Instead of training a model directly on raw gaze coordinates, they used a powerful "teacher" model (Qwen3.5-122B). This teacher was given the image, the question, and the human gaze points. It was tasked with generating a reasoning trace where it selects the most relevant gaze points to explain its logic.
- Supervised Fine-Tuning (SFT): A smaller, more efficient VLM (Qwen3-VL-8B) was trained to mimic these high-quality, gaze-informed reasoning traces.
- Reinforcement Learning (RL): Finally, they applied RL (using the GSPO algorithm) to optimize for task accuracy. Crucially, the RL stage did not use gaze data, yet it further improved the model's alignment with human gaze by reinforcing the effective exploration strategies learned during SFT.
Datasets
The study utilized a mix of newly collected data and established benchmarks:
- Human Gaze Dataset (New): 7,109 trials across 16 different source datasets, including
Visual Probe,PixMo,TallyQA, andChartQA. - Evaluation Benchmarks: The models were tested on a diverse suite including ChartQA and ChartQA-Pro (reasoning), CountBenchQA and CountQA (counting), and MME-RealWorld-Lite and Visual Probe (visual search).
Key Insights and Ideas
- Prior over Sequence: The most critical finding is that gaze is most effective when used as a flexible spatial prior rather than a strict sequence to be imitated. Models forced to explain every human fixation (the "Gaze-All" condition) performed worse than models allowed to selectively use gaze points (GGR).
- RL as an Alignment Tool: Surprisingly, optimizing for accuracy via RL actually increased the model's alignment with human gaze. This suggests that human-like exploration is fundamentally a high-reward strategy for solving complex visual problems.
- Exploration vs. Verification: Behavioral analysis revealed that gaze-trained models (GGR) act more like humans: they spend more time exploring the image before committing to an answer. In contrast, standard models often jump to a conclusion quickly and spend their "thinking" time verifying that conclusion post-hoc.
Key Results
- Superior Alignment: GGR-RL achieved a gaze alignment score of 1.15 bits/fix, significantly higher than the "No-Gaze-RL" baseline (0.99 bits/fix).
- Strong Performance: GGR-RL matched or exceeded the accuracy of the strongest baselines across all tasks, achieving an overall accuracy of 68.8% on the evaluation suite.
- Reliable Grounding: Human evaluators and GPT-4o-based metrics confirmed that GGR's reasoning claims were more likely to be supported by the image regions it pointed to. Specifically, GGR-RL produced "supported claims + points on/near target" in 74.9% of cases, compared to 70.5% for the no-gaze version.
Limitations and Future Work
- Architecture Specificity: The study primarily focused on the Qwen-VL architecture. It remains to be seen if these gaze-transfer benefits hold equally across different VLM families (e.g., Flamingo-style vs. LLaVA-style).
- Biological Divergence: AI models do not have a fovea (the high-resolution center of the human eye) or the same working memory constraints as humans. The authors suggest that future work could explore adding these "biologically-plausible" constraints to VLMs to see if it further improves reasoning.
- Scaling: While 7,000 trials is a significant dataset, it is small compared to the massive web-scale datasets used for general VLM training. Scaling gaze-data collection remains a logistical challenge.
Summary updated