TL;DR
ComboBench is a benchmark evaluating whether Large Language Models (LLMs) can translate high-level gameplay goals into precise, hardware-level Virtual Reality (VR) device manipulations. Evaluating 12 leading LLMs across 262 scenarios from four diverse VR titles reveals that while models excel at high-level task decomposition, they severely struggle with procedural ordering, spatial awareness, and fine-grained motor mapping compared to humans.
What Problem Does This Paper Address?
In embodied interactive environments like VR, human players intuitively bridge the gap between abstract intent (e.g., "tame the horse" or "surrender to the combine guard") and physical hardware inputs (e.g., tilting an HMD, depressing specific controller grips, and pressing buttons in a strict temporal order).
While LLMs increasingly plan tasks in robotics or digital web environments, VR occupies a distinct middle ground: it requires reasoning across continuous 3D space, physics-based affordances, and strict sequential timing—all without the physical safety hazards or slow iteration cycles of physical robots. ComboBench investigates whether state-of-the-art LLMs possess the underlying cognitive abilities to reason about and output these grounded physical device manipulations given purely text-based goal descriptions.
Methods
To systematically dissect model reasoning, the authors partnered with cognitive science experts to define a taxonomy of six core cognitive capabilities:
- Task decomposition (breaking goals into sub-tasks)
- Procedural reasoning (understanding causal and temporal sequence dependencies)
- Spatial reasoning & contextual awareness (interpreting 3D layouts and object states)
- Object interaction & tool-use understanding (mapping virtual affordances to physical controls)
- Motor action mapping & VR transfer (translating abstract actions into device inputs)
- Judgment of termination/continuation conditions (knowing when a loop or action finishes)
Models receive game context and controller guides, outputting step-by-step manipulations in structured JSON. The benchmark measures performance using:
- Strict Step-by-Step Matching (SSM): Requires exact sequence length and high cosine similarity ($> 0.8387$ using
text-embedding-3-large) for every step. - Normalized Step Alignment Score (NSAS): Evaluates common subsequence alignment while penalizing missing or extraneous steps.
- Sequential Order Preservation (SOP): Measures whether matched steps respect correct ordinal execution.
- Semantic Step Coverage (SSC): Measures the fraction of generated steps that match ground-truth steps regardless of order.
- Multi-Path Step Matching (MP-SSM): Accommodates multiple valid human-authored action sequences.
Datasets
The authors introduce ComboBench, comprising 262 human-annotated scenarios curated from comprehensive walkthroughs and gameplay across four distinct VR titles:
- Vivecraft (Minecraft VR; open-world sandbox, crafting, discrete block interactions)
- Half-Life: Alyx (first-person physics shooter, realistic puzzle-solving)
- Moss: Book II (third-person puzzle-platformer, camera-relative controls)
- Into the Radius (survival shooter with complex physical inventory and weapon handling)
Each scenario was played and validated on Oculus Quest 2 hardware by experienced players, aligned with gameplay timestamps, and labeled across the six cognitive capabilities using a human-verified LLM pipeline.
Key Insights and Ideas
- The "Decomposition vs. Sequencing" Disconnect: Models consistently achieve high step alignment (NSAS frequently $> 0.80\text{--}0.92$), demonstrating they know which actions are needed. However, their Sequential Order Preservation (SOP) is remarkably poor ($< 0.30$), proving they struggle to understand prerequisites (e.g., attempting to aim a weapon before completing the pickup action).
- The Sandbox Overconfidence Paradox: In Vivecraft, models attained their highest step alignment scores (up to $0.938$) but abysmal order preservation ($< 0.20$). Familiar, discrete mechanics appear to inflate lexical coverage while obscuring strict procedural order.
- Cognitive Strengths vs. Embodied Deficits: Across models, task decomposition scores rival humans ($7.8\text{--}8.5/10$ vs. human $8.2/10$). Conversely, motor action mapping collapses ($0.5\text{--}4.5/10$).
- Few-Shot Prompting Boosts Structure, Not Grounding: 5-shot prompting improves SOP by $10\text{--}20\times$ over near-zero zero-shot baselines, but exact step replication (SSM) remains low ($< 12\%$), showing examples teach procedural formatting rather than true physical grounding.
Key Results
- Overall LLM Leaders: In 5-shot evaluations, Gemini-3-Pro achieved the top overall performance (NSAS: $0.920$, SOP: $0.214$, SSC: $0.515$), leading in three of the four games. Claude-Sonnet-4.5 (NSAS: $0.915$) and Grok-4 (NSAS: $0.911$, Alyx SOP: $0.351$) also demonstrated competitive generalist capabilities.
- Open-Source Scaling: Parameter scaling yielded massive gains: LLaMA-3-70B raised average NSAS to $0.916$ and SSC to $0.531$ (up from $0.794$ and $0.132$ on LLaMA-3-8B), matching or exceeding several proprietary baselines.
- Human vs. Model Comparison: When humans were tested under identical text-only memory recall constraints, they scored NSAS $0.833$ and SOP $0.122$. While advanced LLMs matched or exceeded humans on textual recall metrics, humans significantly outperformed all models on spatial reasoning ($8.3$ vs. $7.5$ for Gemini-1.5-Pro) and termination condition judgments ($6.5$ vs. $6.0$, $p < 0.05$). Under multi-path evaluation (MP-SSM), human NSAS jumped to $0.931$.
Limitations and Future Work
- Text-Only Modality: Models evaluated only textual descriptions without real-time visual, proprioceptive, or physics feedback, isolating symbolic reasoning from interactive closed-loop control.
- Diminishing Returns with Prompting: Performance gains plateaued sharply after 3–5 examples, indicating fundamental architectural bottlenecks in temporal reasoning that prompting alone cannot resolve.
- Future Directions: Incorporating multimodal inputs (visual scene understanding alongside controller text), integrating simulated embodiment environments for continuous feedback, and explicitly pre-training models on causal physical sequences.
Summary updated