GazeLLM: a plug-and-play zero-shot LLM reasoning framework for boosting gaze target detection

Yaokun Yang, Feng LuVisual 2025 · DOI

TL;DR

GazeLLM is a plug-and-play framework that boosts gaze target detection by using Large Language Models (LLMs) to provide high-level semantic reasoning. By converting visual scenes into structured 3D text descriptions, it allows LLMs to reason about human intent—improving performance on challenging cases by up to 34% without any retraining.

What Problem Does This Paper Address?

Gaze target detection—identifying exactly what or whom a person is looking at in an image—is a cornerstone of understanding human behavior. Currently, the field is split between two imperfect approaches:

  1. Vision-based models: These are great at geometric "pattern matching" (e.g., aligning head orientation with a vector), but they lack "common sense." They might fail to realize a tennis player is likely looking at the ball rather than a random spectator in the background.
  2. Vision-Language Models (VLMs): While these understand context, they struggle with precise spatial localization. They can describe a scene beautifully but often fail to pinpoint the exact coordinates of a gaze target.

The authors identify a "semantic gap": existing models overlook the high-level relationships that govern human attention. GazeLLM aims to bridge this gap by using the reasoning power of text-only LLMs to interpret the why behind a gaze.

Methods

GazeLLM operates as a zero-shot, "plug-and-play" booster for existing vision models. The pipeline consists of three main stages:

  • Structured Object Extraction: Instead of feeding raw pixels to the LLM, the framework uses off-the-shelf tools (MM-GroundingDINO for objects, GazeOnce for 3D gaze direction, and Depth Anything V2 for metric depth) to build a 3D "map" of the scene. This is converted into a structured text dictionary containing object categories, 3D locations, and head orientations.
  • Autonomous Chain-of-Thought (CoT) Reasoning: The framework prompts a pretrained LLM (like ChatGPT o3-mini-high) to perform spatial-semantic analysis. The LLM is guided to "think" through the scene: Which objects are in the person's field of view? What activity are they performing? Which object is most relevant to that activity?
  • Cross-Modal Fusion: The LLM outputs a probability dictionary for potential targets. This is transformed into a Gaussian-weighted "multi-hot" heatmap and multiplied with the heatmap from a traditional vision-based model. This fusion ensures the final prediction is both geometrically plausible and semantically logical.

Datasets

The authors evaluated GazeLLM across several standard benchmarks:

  • GazeFollow & VideoAttnTarget: The primary benchmarks for single-person gaze target detection.
  • VideoCoAtt: Used for Shared Attention (SA) tasks (multiple people looking at the same thing).
  • UCO-LAEO & AVA-LAEO: Used for "Look-At-Each-Other" (LAEO) detection.

Notably, the authors introduced additional object-level ground truths for the GazeFollow test set to specifically measure how well models identify the category of the object being viewed, rather than just a coordinate in space.

Key Insights and Ideas

  • Text as a Clean Interface: By converting the image into structured text, the authors eliminate "visual noise." The LLM doesn't get distracted by lighting or textures; it focuses purely on the spatial and semantic logic of the scene.
  • The Power of Priors: LLMs possess vast latent knowledge about human social norms and activities (e.g., "people playing frisbee usually look at the frisbee"). GazeLLM taps into this without needing a single line of task-specific training data.
  • Interpretability by Design: Because the LLM uses Chain-of-Thought reasoning, the system provides a natural language explanation for its prediction (e.g., "Person 1 is likely looking at the soccer ball because they are in a kicking stance").
  • Boosting, Not Replacing: The framework is designed to sit on top of existing State-of-the-Art (SOTA) models, acting as a "semantic filter" that corrects errors made by purely geometric models.

Key Results

  • Massive Gains on Hard Cases: On challenging scenarios—such as long-range targets or rare object categories—GazeLLM improved the performance of SOTA models by 17% to 34%.
  • Social Gaze Superiority: In multi-person tasks, the framework achieved a 42% Average Precision (AP) gain on the AVA-LAEO benchmark.
  • Outperforming VLMs: Despite not "seeing" the image directly, GazeLLM's text-based reasoning outperformed advanced VLMs (like InternVL2.5) in precise object-level gaze prediction.
  • Zero-Shot Accuracy: The LLM reasoning module alone achieved over 86% Top-1 accuracy in predicting the correct object category from the extracted scene data.

Limitations and Future Work

  • Latency: LLM inference is computationally expensive. The authors note a latency of approximately 300ms per sample, which may be too slow for real-time robotics without further optimization or model distillation.
  • Dependency on Detectors: If the initial object detector (GroundingDINO) fails to find an object, the LLM cannot reason about it. The model also struggles with "no-object" targets, such as a person staring blankly at the sky or a wall.
  • Future Directions: The authors suggest exploring LLM compression for edge deployment and improving the framework's ability to handle occluded targets or non-object gaze points.

Summary updated

Abstract

Abstract In the gaze behavior understanding task, existing vision-based models demonstrate inherent limitations in high-dimensional semantic understanding, while vision-language models (VLMs) encounter challenges in precise object localization. To address this issue, we propose GazeLLM, the first zero-shot large language model (LLM) boosted framework for gaze target reasoning. Our key innovations include three aspects. First, we have structured object extraction. Using off-the-shelf detectors (e.g., MM-GroundingDINO and Depth Anything V2), we convert images into 3D object representations, including head and gaze direction, object categories, and metric depth. Second, we implemented an autonomous chain-of-thought (CoT) reasoning system. We designed self-generated CoT prompts to guide pretrained LLMs, such as ChatGPT o3-mini-high, to predict gaze targets via spatial-semantic analysis. Third, we proposed a plug-and-play module. We employed a novel cross-modal fusion mechanism that combines the LLM’s probability dictionaries with vision-based gaze heatmaps via Gaussian-weighted multi-hot mapping. Extensive experiments show that GazeLLM significantly improves state-of-the-art models, increasing their performance from 17% to 34% on challenging cases, such as long-range targets or rare categories, without the need for retraining. It also extends seamlessly to multi-person social gaze tasks (e.g., a 42% LAEO AP gain on the AVA-LAEO benchmark). Our framework demonstrates superior generalizability and interpretability compared to VLMs, validating the efficacy of LLMs in understanding gaze behavior by mining semantic cues.
BibTeX
@inproceedings{yang2025gazellm,
  author = {Yaokun Yang and Feng Lu},
  title = {{GazeLLM: a plug-and-play zero-shot LLM reasoning framework for boosting gaze target detection}},
  booktitle = {Visual},
  year = {2025},
  doi = {10.1007/s44267-025-00101-1},
}
Figure