Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

Peng Chen, Pi Bu, Jun Song, Yuan Gao, Bo ZhengSubmission to ICLR 2025 · Rejected

TL;DR

Can vision-language models (VLMs) tackle complex, fast-paced action role-playing games without relying on internal game APIs? Using the hit AAA title Black Myth: Wukong as a testbed, the authors introduce VARP (Vision Action Role-Playing), an agent framework that pairs visual perception with modular action planning and human trajectory retrieval. VARP succeeds in roughly 90% of easy-to-medium combat encounters and demonstrates that multimodal foundation models can execute coordinated combat combos purely from screenshots.

What Problem Does This Paper Address?

Most LLM- and VLM-based game agents operate under relaxed conditions: they either interface directly with game engines via dedicated APIs (e.g., Voyager in Minecraft) or rely heavily on explicit on-screen text instructions (e.g., Cradle in Red Dead Redemption 2).

In contrast, Action Role-Playing Games (ARPGs) like Black Myth: Wukong present two steep hurdles:

  1. No textual crutches or open APIs: The UI provides minimal text guidance; players must perceive boss telegraphs, health bars, and spatial orientation directly from raw pixels.
  2. Dynamic, tightly timed combat: Unlike sandbox games, ARPGs punish delayed or misplaced actions. Prior autonomous approaches in ARPGs relied on reinforcement learning (RL), which suffers from brittle generalization and demands millions of environment interactions per scenario.

This paper investigates the upper capability boundaries of off-the-shelf VLMs (such as GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro) in closed-loop, visual-only ARPG combat and navigation.

Methods

To bridge high-level multimodal reasoning and millisecond-level gameplay execution, the authors developed the VARP agent:

Screen Capture (Photo Mode Pause) 
  → Multi-VLM Modules (OCR + Grounding DINO + DTSA Sub-modules)
  → Action Combo Synthesis / Retrieval (Action Library & SOAG)
  → Executable Python Scripts (Simulated Mouse & Keyboard)

The framework is built around two primary systems:

  • Action Planning System:
  • Decomposable Task-Specific Auxiliary (DTSA): Rather than prompting a single VLM with overwhelming context, DTSA splits decision-making into five parallel sub-modules: Enemy Status, Combat Method (light vs. heavy attack charges), Health Monitoring, Spell Cooldowns, and an Integration Sub-module that selects the final action.
  • Self-Optimizable Action Generation (SOAG): Synthesizes new combat routines on the fly. By analyzing sequential frames of enemy animations (e.g., identifying a 3-hit axe swing), SOAG compiles counter-combos (e.g., sequence of 3 dodges followed by light attacks) and stores them as reusable Python functions.
  • Human-Guided Trajectory System: For long-horizon goals like pathfinding, the agent captures the current scene, queries a database of human gameplay via visual similarity embeddings, and prompts a VLM to convert the retrieved expert trajectories into actionable wayfinding macros.
  • Inference Mechanism: Because current frontier VLMs cannot process 60 FPS real-time streams with zero latency, the system pauses the game using built-in photo mode during inference before firing scripted Python keyboard and mouse events.

Datasets

  • Human Operation Dataset: 1,000 recorded video clips paired with timestamped keyboard and mouse logs collected from 200 human players across Chapter 1 of Black Myth: Wukong. A subset of optimal runs is categorized as "clean" data by pruning redundant inputs.
  • BMW Benchmark: 13 standardized tasks categorized by difficulty:
  • Easy (Tasks 1–8): Basic mechanics, gathering, and standard mob skirmishes (e.g., WolfScout, WolfSoldier).
  • Middle (Task 9): Crow Diviner combat.
  • Hard (Task 10): Bullguard mini-boss encounter.
  • Very Hard (Tasks 11–13): Major boss fights (Guangzhi, Wandering Wight) and a 5-minute autonomous 3D navigation task through terrain containing invisible boundaries.

Key Insights and Ideas

  • Context decomposition prevents agent blindness: When monolithic prompts handle visual parsing, cooldowns, and tactical choices simultaneously, VLMs frequently hallucinate or forget simple tasks (e.g., attempting to heal at 100% health). DTSA's parallelized architecture cleanly eliminates these errors.
  • Refining combos over time: SOAG iteratively optimizes counter-actions. For example, against the Bullguard, early actions simply spammed consecutive dodges before counter-attacking; over iterations, the agent learned to weave strikes into the recovery frames between dodges.
  • Agent conciseness vs. human spam: In sustained fights, novice human players execute dozens of redundant, panicked button inputs. In contrast, the VARP agent averaged significantly fewer, more concise atomic inputs per encounter.
  • Zero-shot 3D navigation is a failure mode: Without explicit minimaps or waypoint markers, VLMs achieved a 0% success rate navigating complex 3D geometry due to depth ambiguity and invisible walls. Introducing trajectory retrieval jumped this success rate to 40%.

Key Results

  • Basic to Medium Combat (Tasks 1–9): VARP achieved near 100% success on Tasks 1–8 and a 40–60% success rate on Task 9 across tested VLMs, rivaling novice humans.
  • Hard Encounters (Task 10 - Bullguard): VARP driven by GPT-4o achieved a 40% success rate, outperforming novice humans (15.6%) and completely outclassing the adapted baseline Cradle framework (0%).
  • Very Hard Encounters (Task 11 - Guangzhi): While neither human novices nor VARP could reliably defeat the boss in single-session evaluations (0% clear rate), VARP successfully depleted ~40% of the boss's HP pool on average through purely visual prompt-action loops.

Limitations and Future Work

  • Reliance on game pauses: The system depends on game-pausing (photo mode) to accommodate multi-second VLM API round-trip latencies, leaving true real-time, 60 FPS combat unaddressed.
  • Spatial perception bottlenecks: Pure 2D vision models struggle with 3D spatial geometry, obstacle navigation, and non-visual game constraints without expert trajectory demonstrations.
  • Scope: Benchmarks were confined to Chapter 1 encounters in a single title.
  • Future Work: The authors highlight the need for domain-specific, low-latency video models (e.g., VideoGameBunny) and integrating imitation learning with multi-modal retrieval-augmented generation (RAG).

Summary updated

Abstract

Recently, large language model (LLM)-based agents have made significant advances across various fields. One of the most popular research areas involves applying these agents to video games. Traditionally, these methods have relied on game APIs to access in-game environmental and action data. However, this approach is limited by the availability of APIs and does not reflect how humans play games. With the advent of vision language models (VLMs), agents now have enhanced visual understanding capabilities, enabling them to interact with games using only visual inputs. Despite these advances, current approaches still face challenges in action-oriented tasks, particularly in action role-playing games (ARPGs), where reinforcement learning methods are prevalent but suffer from poor generalization and require extensive training. To address these limitations, we select an ARPG, ``Black Myth: Wukong'', as a research platform to explore the capability boundaries of existing VLMs in scenarios requiring visual-only input and complex action output. We define 13 tasks within the game, with 76.9% focusing on combat, and incorporate several state-of-the-art VLMs into this benchmark. Additionally, we will release a human operation dataset containing recorded gameplay videos and operation logs, including mouse and keyboard actions. Moreover, we propose a novel VARP (Vision Action Role-Playing) agent framework, consisting of an action planning system and a human-guided trajectory system. Our framework demonstrates the ability to perform basic tasks and succeed in 90% of easy and medium-level combat scenarios. This research aims to provide new insights and directions for applying multimodal agents in complex action game environments. The code and datasets will be made available at https://varp-agent.github.io/.
BibTeX
@inproceedings{chen2024can,
  author = {Peng Chen and Pi Bu and Jun Song and Yuan Gao and Bo Zheng},
  title = {{Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case}},
  booktitle = {OWA @ NeurIPS},
  year = {2024},
  url = {https://openreview.net/forum?id=9Aoy1AsHaQ},
}
Figure