Game Generation via Large Language Models

Chengpeng Hu, Yunlong Zhao, Jialin Liu

Abstract

Recently, the emergence of large language models (LLMs) has unlocked new opportunities for procedural content generation. However, recent attempts mainly focus on level generation for specific games with defined game rules such as Super Mario Bros. and Zelda. This paper investigates the game generation via LLMs. Based on video game description language, this paper proposes an LLM-based framework to generate game rules and levels simultaneously. Experiments demonstrate how the framework works with prompts considering different combinations of context. Our findings extend the current applications of LLMs and offer new insights for generating new games in the area of procedural content generation.

Introduction

Procedural Content Generation (PCG) has traditionally relied on machine learning models trained on specific, structured data, making them rigid and difficult to generalize across different game genres. While LLMs have recently been used to "prompt" new levels for games like Super Mario Bros., these attempts assume the game rules are already fixed.

The authors identify a significant gap: generating the game logic itself. Creating a game requires a coherent relationship between the rules (how things interact) and the level design (where things are placed). Until now, automating the simultaneous creation of both via natural language has been largely unexplored.

The LLMGG Framework

The authors propose the LLMGG (Large Language Model Game Generation) framework, which uses VGDL (Video Game Description Language) as the intermediate representation. VGDL is a human-readable, high-level language that defines four key components:

  1. SpriteSet: The objects in the game (e.g., avatars, walls, goals).
  2. LevelMapping: How characters in a text file map to those objects.
  3. InteractionSet: What happens when objects collide (e.g., avatar wall > stepBack).
  4. TerminationSet: The win/loss conditions.
Overview of the LLMGG framework for simultaneous rule and level generation.

The core of the methodology lies in Prompt Design. The researchers tested seven different prompt configurations ($P_1$ to $P_7$) to see how much "context" an LLM needs to succeed. These contexts ranged from simple instructions to providing the full VGDL grammar, specific type constraints, and few-shot examples (using the game Aliens as a template).

Experimental Setup

The study evaluates the zero-shot and few-shot capabilities of three pre-trained models: GPT-3.5, GPT-4, and Gemma 7B. The evaluation benchmark consisted of 10 trials per prompt for each model, tasked with generating a functional "Maze" game. The generated outputs were validated using the GVGAI (General Video Game Artificial Intelligence) gym engine to check for:

  • Parsability: Is the code syntactically correct?
  • Logic: Does the game have a win condition and necessary interactions?
  • Mappability: Do the level characters actually match the defined sprites?

Results and Analysis

The experimental results highlight a clear hierarchy in model performance and the critical role of prompt engineering.

Performance comparison of LLMs across different prompt configurations.

Key Findings

  • GPT-4 Dominance: GPT-4 was the only model capable of generating 10/10 perfectly "Correct" games, specifically when using prompt $P_7$.
  • GPT-3.5 Struggles: Even with the best prompts, GPT-3.5 peaked at a 40% success rate, frequently failing on logic or mapping.
  • Gemma 7B Failure: The smaller Gemma model failed to generate a single "Correct" game across all prompts, highlighting a significant reasoning gap in smaller open-source models for formal language generation.
  • Prompt Sensitivity: The jump from prompt $P_4$ (basic grammar) to $P_5$ (adding specific interaction constraints) saw a massive improvement in logical consistency for GPT-4 (from 0/10 to 8/10 correct).

The "Word Order" Hallucination

One of the most fascinating findings is why LLMs fail at game logic. In VGDL, the interaction goal avatar > killSprite means the goal is removed. However, LLMs often assume a natural language "Subject-Verb-Object" (SVO) logic, thinking the avatar should be the one affected. This conflict between natural language intuition and formal syntax is a major hurdle for zero-shot game design.

Discussion and Future Work

While LLMs show promise, they face a "Complexity Ceiling." The experiments were limited to a simple "Maze" game, and it remains to be seen if they can handle complex, multi-state logic required for RPGs. Furthermore, a "Self-Correction Paradox" was observed: when pointed out logical errors, models could explain them in plain English but often failed to fix the underlying code.

Future directions include extending this framework to 3D game generation and fine-tuning LLMs specifically on VGDL datasets to mitigate syntax-related hallucinations. Human intervention remains necessary to refine complex mechanics and ensure the resulting games are "fun."

Summary updated

BibTeX
@inproceedings{hu2024game,
  author = {Chengpeng Hu and Yunlong Zhao and Jialin Liu},
  title = {{Game Generation via Large Language Models}},
  booktitle = {CoG},
  year = {2024},
  url = {https://openreview.net/forum?id=VCEVwqYGLG},
}
Figure