Abstract
This paper introduces ViRAC, a novel framework that leverages Vision-Language Models (VLMs) and Large Language Models (LLMs) to generate human-like head movements for virtual agents. By transitioning from traditional saliency-based methods to a reasoning-driven approach, ViRAC enables agents to prioritize safety, curiosity, and social norms. Our results demonstrate that this cognitive-first approach produces behavior that is significantly more realistic and context-aware than existing data-driven baselines.
Introduction
In virtual reality and gaming, creating "believable" agents is essential for maintaining user immersion. A critical aspect of this believability is how an agent moves its head to gather information. Historically, agents have relied on data-driven models or saliency-based methods that focus on visual features like brightness or motion. However, these methods lack cognitive context; they fail to capture why a human looks at a specific object, such as checking a crosswalk for safety or scanning a mall for an escalator. ViRAC addresses this gap by treating head movement as a continuous loop of perception and reasoning.
The ViRAC Framework
The ViRAC (Vision-Reasoning Agent Head Movement Control) framework consists of three primary components designed to emulate human cognitive processes.
Perception and Memory
The framework utilizes a Vision-Language Model (VLM) to analyze the agent's first-person view, identifying objects and generating textual descriptions. These descriptions are stored in a Foundational Memory Module (FMM), allowing the agent to maintain awareness of objects even when they are no longer in its immediate field of view.
Decision-Making and Cognitive Goals
An LLM serves as the decision-making core, processing high-level goals, memory, and action history. Through Cognitive Goal Decomposition (CGD), the LLM breaks down objectives into sub-tasks based on three pillars:
- Safety: Identifying and avoiding threats.
- Curiosity: Exploring the environment to gain information.
- Social Schema: Adhering to human social norms, such as checking both ways at a street crossing.
Experimental Evaluation
To validate ViRAC, the researchers developed a Human Rationale Dataset by tracking 20 participants in various VR environments (e.g., Mall, Café, Bus). Participants provided self-reported rationales for their head movements, creating a ground-truth for reasoning. Scenarios were split into Minimal-Distraction (MDC) and Attention-Provoking (APC) conditions. ViRAC was compared against Track, a state-of-the-art saliency-based model.
Key Results and Discussion
ViRAC demonstrated superior performance in both objective and subjective metrics:
- Trajectory Accuracy: Using Dynamic Time Warping (DTW), ViRAC’s head rotations were significantly closer to human data than the baseline across almost all scenarios (Mall, Street, Crossing, Café).
- Subjective Realism: In user studies, ViRAC achieved scores comparable to real human data in naturalness, safety, and curiosity, consistently outperforming saliency-based models.
- Reasoning Nuance: The model successfully distinguished between "confirmation" (verifying known objects) and "exploration" (searching for new information).
One notable finding was the "Santa Claus" limitation. In a bus scenario, the baseline model outperformed ViRAC because a "Santa Claus" character distracted human participants. The LLM, being highly logical, did not prioritize looking at the character, suggesting that future iterations should account for "visual novelty" or "oddities" that naturally attract human attention even when they are not task-relevant.
Conclusion
ViRAC represents a significant shift toward reasoning-driven agent behavior. While currently limited to visual inputs and decoupled from body locomotion, future work aims to integrate multimodal inputs (audio, haptics) and couple head movements with path planning to further enhance agent believability in complex virtual worlds.
Summary updated