TL;DR
GazeLLM is a plug-and-play framework that boosts gaze target detection by using Large Language Models (LLMs) to provide high-level semantic reasoning. By converting visual scenes into structured 3D text descriptions, it allows LLMs to reason about human intent—improving performance on challenging cases by up to 34% without any retraining.
What Problem Does This Paper Address?
Gaze target detection—identifying exactly what or whom a person is looking at in an image—is a cornerstone of understanding human behavior. Currently, the field is split between two imperfect approaches:
- Vision-based models: These are great at geometric "pattern matching" (e.g., aligning head orientation with a vector), but they lack "common sense." They might fail to realize a tennis player is likely looking at the ball rather than a random spectator in the background.
- Vision-Language Models (VLMs): While these understand context, they struggle with precise spatial localization. They can describe a scene beautifully but often fail to pinpoint the exact coordinates of a gaze target.
The authors identify a "semantic gap": existing models overlook the high-level relationships that govern human attention. GazeLLM aims to bridge this gap by using the reasoning power of text-only LLMs to interpret the why behind a gaze.
Methods
GazeLLM operates as a zero-shot, "plug-and-play" booster for existing vision models. The pipeline consists of three main stages:
- Structured Object Extraction: Instead of feeding raw pixels to the LLM, the framework uses off-the-shelf tools (MM-GroundingDINO for objects, GazeOnce for 3D gaze direction, and Depth Anything V2 for metric depth) to build a 3D "map" of the scene. This is converted into a structured text dictionary containing object categories, 3D locations, and head orientations.
- Autonomous Chain-of-Thought (CoT) Reasoning: The framework prompts a pretrained LLM (like ChatGPT o3-mini-high) to perform spatial-semantic analysis. The LLM is guided to "think" through the scene: Which objects are in the person's field of view? What activity are they performing? Which object is most relevant to that activity?
- Cross-Modal Fusion: The LLM outputs a probability dictionary for potential targets. This is transformed into a Gaussian-weighted "multi-hot" heatmap and multiplied with the heatmap from a traditional vision-based model. This fusion ensures the final prediction is both geometrically plausible and semantically logical.
Datasets
The authors evaluated GazeLLM across several standard benchmarks:
- GazeFollow & VideoAttnTarget: The primary benchmarks for single-person gaze target detection.
- VideoCoAtt: Used for Shared Attention (SA) tasks (multiple people looking at the same thing).
- UCO-LAEO & AVA-LAEO: Used for "Look-At-Each-Other" (LAEO) detection.
Notably, the authors introduced additional object-level ground truths for the GazeFollow test set to specifically measure how well models identify the category of the object being viewed, rather than just a coordinate in space.
Key Insights and Ideas
- Text as a Clean Interface: By converting the image into structured text, the authors eliminate "visual noise." The LLM doesn't get distracted by lighting or textures; it focuses purely on the spatial and semantic logic of the scene.
- The Power of Priors: LLMs possess vast latent knowledge about human social norms and activities (e.g., "people playing frisbee usually look at the frisbee"). GazeLLM taps into this without needing a single line of task-specific training data.
- Interpretability by Design: Because the LLM uses Chain-of-Thought reasoning, the system provides a natural language explanation for its prediction (e.g., "Person 1 is likely looking at the soccer ball because they are in a kicking stance").
- Boosting, Not Replacing: The framework is designed to sit on top of existing State-of-the-Art (SOTA) models, acting as a "semantic filter" that corrects errors made by purely geometric models.
Key Results
- Massive Gains on Hard Cases: On challenging scenarios—such as long-range targets or rare object categories—GazeLLM improved the performance of SOTA models by 17% to 34%.
- Social Gaze Superiority: In multi-person tasks, the framework achieved a 42% Average Precision (AP) gain on the AVA-LAEO benchmark.
- Outperforming VLMs: Despite not "seeing" the image directly, GazeLLM's text-based reasoning outperformed advanced VLMs (like InternVL2.5) in precise object-level gaze prediction.
- Zero-Shot Accuracy: The LLM reasoning module alone achieved over 86% Top-1 accuracy in predicting the correct object category from the extracted scene data.
Limitations and Future Work
- Latency: LLM inference is computationally expensive. The authors note a latency of approximately 300ms per sample, which may be too slow for real-time robotics without further optimization or model distillation.
- Dependency on Detectors: If the initial object detector (GroundingDINO) fails to find an object, the LLM cannot reason about it. The model also struggles with "no-object" targets, such as a person staring blankly at the sky or a wall.
- Future Directions: The authors suggest exploring LLM compression for edge deployment and improving the framework's ability to handle occluded targets or non-object gaze points.
Summary updated