GameDevBench: Evaluating Agentic Capabilities Through Game Development

Wayne Chi, Yixiong Fang, Arnav Yayavaram, Siddharth Yayavaram, Seth Karten, Qiuhong Anna Wei, Runkun Chen, Alexander Wang, Valerie Chen, Ameet Talwalkar, Chris Donahue

TL;DR

GameDevBench is a new multimodal benchmark that challenges AI agents to develop games using the Godot engine. By requiring agents to navigate complex codebases while simultaneously managing visual assets like sprites and shaders, the benchmark reveals a significant "multimodal gap": even the best agents solve only about 53.8% of tasks, with performance dropping sharply on visually demanding graphics work.

What Problem Does This Paper Address?

While AI "coding agents" have become remarkably proficient at text-based software engineering, they often struggle when the job requires understanding the physical or visual world. Most existing benchmarks are unimodal (text-only) or focus on simple web UI generation.

Game development represents a "final boss" for agentic capabilities. It requires multimodal reasoning (connecting code to animations, collision boxes, and shaders), context-rich navigation (managing hundreds of files and assets), and temporal understanding (verifying that a character moves correctly over time). The authors argue that until agents can bridge the gap between "writing code" and "understanding the visual result," they won't be ready for complex, real-world engineering.

Methods

The researchers developed a "continually renewable" pipeline to generate tasks:

  1. Data Sourcing: They scraped 57 YouTube tutorials and 31 web tutorials for the Godot 4 engine, extracting transcripts and corresponding GitHub repositories.
  2. Automated Construction: Using a "Codex Agent," they decomposed these tutorials into 333 discrete tasks.
  3. Human-in-the-loop Refinement: Eight human annotators verified the tasks for solvability and categorized them by difficulty and skill set.
  4. Deterministic Evaluation: Unlike many multimodal benchmarks that rely on subjective "LLM-as-a-Judge" scoring, GameDevBench uses Godot’s internal scripting to run unit tests. It checks the actual state of the game engine (e.g., "Is the collision box at the right coordinates?") to provide objective pass/fail results.
  5. Visual Feedback Loops: The authors introduced two tools to help agents: an MCP (Model Context Protocol) server that provides editor screenshots and a runtime video recorder so the agent can "see" the game in motion.

Datasets

The primary contribution is GameDevBench, which consists of:

  • 333 tasks (154 core tasks and 179 variants).
  • Four skill categories: 2D Graphics & Animation, 3D Graphics & Animation, Gameplay Logic, and User Interface.
  • High Complexity: The average solution requires three times more lines of code and file changes than the popular SWE-Bench, making it one of the most demanding software benchmarks to date.

Key Insights and Ideas

  • The Multimodal Tax: There is a clear correlation between visual complexity and agent failure. Agents performed reasonably well on "Gameplay Logic" (51.4% success) but struggled significantly with "2D Graphics" (33.0%), where they often picked the wrong sprite from a sheet or misconfigured shader parameters.
  • Structural Failures: A common error pattern was "mis-parenting" nodes. In Godot, the hierarchy of the scene tree is vital. Agents often understood what node to add but put it in the wrong place in the tree, causing the game to break.
  • The Power of "Seeing": Providing visual feedback (screenshots and video) consistently boosted performance. For example, GPT-5.4’s success rate jumped from 41.1% to 52.0% simply by being allowed to look at the editor state.
  • Frontier Gap: The performance gap between top-tier models (Gemini 3 Pro, GPT-5.4) and open-weights models (Kimi K2.5, Qwen) is massive—nearly 50 percentage points—suggesting that multimodal reasoning is currently a "frontier-only" capability.

Key Results

  • Top Performer: Gemini 3 Pro (with visual feedback) achieved the highest score at 53.8%.
  • Baseline vs. Multimodal: GPT-5.4 improved by 10.9 percentage points (a 26.5% relative increase) when given access to screenshots and video.
  • Cost Efficiency: Gemini 3 Flash emerged as the most cost-effective model, offering high performance at a fraction of the cost of GPT-5.4, which saw costs triple when multimodal tools were enabled.
  • Complexity Metric: Reference solutions averaged 4.7 files edited and 114.1 lines of code changed, confirming the benchmark's high difficulty.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Code-Centric Solutions: Currently, agents solve tasks by writing code to modify scene files. Future iterations could evaluate "computer use" agents that interact directly with the Godot GUI.
  • Engine Specificity: The benchmark is currently limited to Godot 4. While Godot is open-source and representative of engines like Unity, expanding to other environments would increase generalizability.
  • Asset Creation: The current tasks focus on using existing assets. The authors suggest that future work could involve agents generating the assets (textures, sounds) themselves.
  • Human Validation: While the pipeline is automated, it still requires minor human validation (about 10 minutes per task) to ensure 100% accuracy in the ground truth.

Summary updated

Abstract

Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex---the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only $53.8$% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from $51.4$% on gameplay-oriented tasks to $33.0$% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from $41.1$% to $52.0$% when given visual feedback.
BibTeX
@inproceedings{chi2026gamedevbench,
  author = {Wayne Chi and Yixiong Fang and Arnav Yayavaram and Siddharth Yayavaram and Seth Karten and Qiuhong Anna Wei and Runkun Chen and Alexander Wang and Valerie Chen and Ameet Talwalkar and Chris Donahue},
  title = {{GameDevBench: Evaluating Agentic Capabilities Through Game Development}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=Iexhb5lL3t},
}
Figure