ViRAC: A Vision-Reasoning Agent Head Movement Control Framework in Arbitrary Virtual Environments

Juyeong Hwang, Seong-Eun Hong, Hyeongyeop KangarXiv 2025 · DOI · arXiv

Abstract

This paper introduces ViRAC, a novel framework that leverages Vision-Language Models (VLMs) and Large Language Models (LLMs) to generate human-like head movements for virtual agents. By transitioning from traditional saliency-based methods to a reasoning-driven approach, ViRAC enables agents to prioritize safety, curiosity, and social norms. Our results demonstrate that this cognitive-first approach produces behavior that is significantly more realistic and context-aware than existing data-driven baselines.

Introduction

In virtual reality and gaming, creating "believable" agents is essential for maintaining user immersion. A critical aspect of this believability is how an agent moves its head to gather information. Historically, agents have relied on data-driven models or saliency-based methods that focus on visual features like brightness or motion. However, these methods lack cognitive context; they fail to capture why a human looks at a specific object, such as checking a crosswalk for safety or scanning a mall for an escalator. ViRAC addresses this gap by treating head movement as a continuous loop of perception and reasoning.

The ViRAC Framework

The ViRAC (Vision-Reasoning Agent Head Movement Control) framework consists of three primary components designed to emulate human cognitive processes.

Perception and Memory

The framework utilizes a Vision-Language Model (VLM) to analyze the agent's first-person view, identifying objects and generating textual descriptions. These descriptions are stored in a Foundational Memory Module (FMM), allowing the agent to maintain awareness of objects even when they are no longer in its immediate field of view.

The VLM identifies key areas like the Arcade Zone to provide textual descriptions for the reasoning engine.

Decision-Making and Cognitive Goals

An LLM serves as the decision-making core, processing high-level goals, memory, and action history. Through Cognitive Goal Decomposition (CGD), the LLM breaks down objectives into sub-tasks based on three pillars:

  • Safety: Identifying and avoiding threats.
  • Curiosity: Exploring the environment to gain information.
  • Social Schema: Adhering to human social norms, such as checking both ways at a street crossing.
The ViRAC framework maps visual perception to specific actions and rationales, emulating human-like prioritization.

Experimental Evaluation

To validate ViRAC, the researchers developed a Human Rationale Dataset by tracking 20 participants in various VR environments (e.g., Mall, Café, Bus). Participants provided self-reported rationales for their head movements, creating a ground-truth for reasoning. Scenarios were split into Minimal-Distraction (MDC) and Attention-Provoking (APC) conditions. ViRAC was compared against Track, a state-of-the-art saliency-based model.

Key Results and Discussion

ViRAC demonstrated superior performance in both objective and subjective metrics:

  • Trajectory Accuracy: Using Dynamic Time Warping (DTW), ViRAC’s head rotations were significantly closer to human data than the baseline across almost all scenarios (Mall, Street, Crossing, Café).
  • Subjective Realism: In user studies, ViRAC achieved scores comparable to real human data in naturalness, safety, and curiosity, consistently outperforming saliency-based models.
  • Reasoning Nuance: The model successfully distinguished between "confirmation" (verifying known objects) and "exploration" (searching for new information).

One notable finding was the "Santa Claus" limitation. In a bus scenario, the baseline model outperformed ViRAC because a "Santa Claus" character distracted human participants. The LLM, being highly logical, did not prioritize looking at the character, suggesting that future iterations should account for "visual novelty" or "oddities" that naturally attract human attention even when they are not task-relevant.

Conclusion

ViRAC represents a significant shift toward reasoning-driven agent behavior. While currently limited to visual inputs and decoupled from body locomotion, future work aims to integrate multimodal inputs (audio, haptics) and couple head movements with path planning to further enhance agent believability in complex virtual worlds.

Summary updated

Abstract

Creating lifelike virtual agents capable of interacting with their environments is a longstanding goal in computer graphics. This paper addresses the challenge of generating natural head rotations, a critical aspect of believable agent behavior for visual information gathering and dynamic responses to environmental cues. Although earlier methods have made significant strides, many rely on data-driven or saliency-based approaches, which often underperform in diverse settings and fail to capture deeper cognitive factors such as risk assessment, information seeking, and contextual prioritization. Consequently, generated behaviors can appear rigid or overlook critical scene elements, thereby diminishing the sense of realism. In this paper, we propose \textbf{ViRAC}, a \textbf{Vi}sion-\textbf{R}easoning \textbf{A}gent Head Movement \textbf{C}ontrol framework, which exploits the common-sense knowledge and reasoning capabilities of large-scale models, including Vision-Language Models (VLMs) and Large-Language Models (LLMs). Rather than explicitly modeling every cognitive mechanism, ViRAC leverages the biases and patterns internalized by these models from extensive training, thus emulating human-like perceptual processes without hand-tuned heuristics. Experimental results in multiple scenarios reveal that ViRAC produces more natural and context-aware head rotations than recent state-of-the-art techniques. Quantitative evaluations show a closer alignment with real human head-movement data, while user studies confirm improved realism and cognitive plausibility.
BibTeX
@misc{hwang2025virac,
  author = {Juyeong Hwang and Seong-Eun Hong and Hyeongyeop Kang},
  title = {{ViRAC: A Vision-Reasoning Agent Head Movement Control Framework in Arbitrary Virtual Environments}},
  year = {2025},
  eprint = {2502.10046},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.10046},
}
Figure