GameWM: Toward Next-Generation Video Games with Interaction-Centric World Models

Bing He, Runze Wang, Fei Zha, Zhongyuan Wu, Ke Xu, Tao Zhuang, Yuxuan Xie, Zhongyuan Liu

TL;DR

GameWM proposes a "division of labor" for next-generation games: a traditional engine handles the deterministic rules and world state, while a generative video model (VM) creates the high-fidelity visuals and fine-grained interaction details. By moving the VM from a simple "renderer" to a source of customized motion, the system allows a single game mechanic to be realized in infinite ways—turning a generic ranged attack into a fireball, an ice lance, or a rifle burst based on a player's unique character sheet.

What Problem Does This Paper Address?

For decades, game development has been a battle against the "content wall." High-fidelity graphics and complex animations are incredibly expensive to author by hand. While recent "world models" (like Sora or Genie) can generate impressive video, they struggle with stability: they "forget" the world layout, fail to follow strict game rules, and can't maintain a consistent character identity across different scenes.

Existing hybrid systems use a game engine to provide a "proxy" (like a depth map) that the VM then "restyles." However, these systems usually treat the VM as a fancy filter. If the engine hasn't already animated a character opening a car door, the VM won't do it either. This leaves the VM’s generative potential untapped and limits gameplay diversity. GameWM aims to bridge this gap by letting the VM handle the fine-grained interactions—the nuanced way a hand grabs a handle or a spell erupts—that are currently too costly to animate for every possible character combination.

Methods

The authors introduce a system architecture built on a clear contract between two components:

  1. The GameWM-engine: A lightweight engine (which could even be written by an AI coding agent) that computes the "truth." It tracks rules, health, collisions, and camera poses. It outputs a coarse proxy video (low-poly meshes, capsules, or depth maps) and a log of events (e.g., "Player A hits Enemy B at frame 80").
  2. The GameWM-VM: A large-scale video model (ranging from 5B to 33B parameters) that takes the coarse proxy and "realizes" it.

Key Innovations: * Interaction-Centric Conditioning: The VM is conditioned on "authored once" assets: a character sheet (front/side/back views) and skill descriptions. This ensures that even if the engine only shows a generic capsule moving, the VM knows to render it as a specific "Fire Mage" performing a "Flame Fist" attack. * Ref2VA Interface: The models use a reference-to-video-and-audio pathway, treating the proxy, character sheet, and first frame as in-context references rather than using separate control modules like ControlNet. * Real-Time Distillation: To make this playable, the authors distilled their 33B "teacher" model into a 4-step causal student model (GameWM-Wan2.2-5B) capable of generating video chunk-by-chunk with minimal latency.

Datasets

The authors built an Interaction-Centric Data Engine to move beyond simple "exploration" data.

  • Sources: They mined clips from GTA V (using internal engine buffers for exact ground truth), ABot-World, CARLA, and Magpie.
  • Monocular Estimation: For titles where they lacked engine access, they used VLMs and monocular depth models to estimate proxies from screen recordings.
  • Curation: Unlike standard datasets that sample frames uniformly, GameWM cuts clips specifically around interactions (entering vehicles, melee strikes, ranged shots).
  • Stylized Targets: They used a high-quality "teacher" model to render multiple stylized versions of the same proxy (e.g., "steampunk" vs. "yokai edo" styles) to teach the model that one proxy can have many valid visual realizations.

Key Insights and Ideas

  • The "Sweet Point" of Stability: The paper argues that game stability collapses when the "world state" moves into the neural network. By keeping the state in the engine but moving the interaction detail to the VM, they maximize generative creativity without losing the ability to have a fair, playable game.
  • Mechanic × Attribute: This is the "aha" moment for scalability. A designer writes a "ranged attack" mechanic once. By fusing it with a character's "fire" attribute, the VM automatically generates a fireball. If the player switches to an "ice" character, the same engine code results in an ice shard, with the VM generating the appropriate new animations and effects on the fly.
  • Identity via First Frame: Interestingly, the researchers found that the "first frame" of a scene carries most of the character's identity. While character sheets help, the model primarily uses the initial authored look of the scene to maintain consistency.

Key Results

  • One Mechanic, Many Realizations: Qualitative results show eight different characters (Vampire, Forest Druid, etc.) performing the same "ranged attack" proxy. Each realization features unique casting poses, projectiles, and impact effects generated entirely by the VM.
  • Consistency: Using DINOv2 similarity scores, they proved the model maintains high character identity across 16 different scene-character combinations.
  • Performance: The 5B causal model, running on a node of eight H200 GPUs, can generate its first chunk of gameplay in 3.1 seconds and then sustain 13–18 FPS. While not yet at the 60 FPS standard for high-end gaming, it demonstrates a path toward real-time neural rendering on consumer hardware (like an RTX 5090).

Limitations and Future Work

  • Model Scaling Issues: Smaller models (5B and 14B) struggle with timing. They sometimes "fire" a projectile a second earlier than the engine proxy dictates, which would break gameplay synchronization.
  • Visual Faintness: In the distilled 4-step models, complex effects like fireballs can appear "faint" or less detailed compared to the 30-step teacher model.
  • Occlusion: Identity consistency drops significantly when the character is viewed from the side or is partially occluded by objects in the world.
  • Future Direction: The authors suggest that the next step is improving the "causal" distillation to ensure smaller models respect the engine's clock as strictly as the larger models do.

Summary updated

Abstract

For more than half a century, video games have pursued better graphics, more natural interaction, and more engaging gameplay. Generative video models (VMs) now bear on all three. Neural networks already draw most of the pixels in shipped games, and recent world models let a lightweight game engine keep the world state while a VM turns the engine's coarse proxy video into final frames. These systems, however, treat the VM as a renderer that only restyles motion the engine has already authored, which leaves most of its generative capacity unused and limits the diversity a game can offer. We present GameWM, a design for next-generation games built on a clear division of labor. The GameWM-engine keeps what must be exact: rules, state, camera, and the layout of the world, which also spares the video model from having to remember it. The GameWM-VM generates what is expensive to author by hand: the look of the world and the detail of the player's interactions. One game mechanic can then be realized in many visually and physically distinct ways, while its outcome stays exactly as the engine computed it. To realize this division of labor, we build GameWM as a complete system: a data engine centered on interactions rather than exploration, an interface through which authored character sheets, scene art, and skill descriptions enter the model, and a model family from 5B to 33B parameters with a real-time causal variant for play. Together they give every player a fresh version of the same game: each player's own character, world, and skills turn the same engine mechanics into a new experience, while every event keeps the timing and outcome the GameWM-engine computed.
BibTeX
@misc{he2026gamewm,
  author = {Bing He and Runze Wang and Fei Zha and Zhongyuan Wu and Ke Xu and Tao Zhuang and Yuxuan Xie and Zhongyuan Liu},
  title = {{GameWM: Toward Next-Generation Video Games with Interaction-Centric World Models}},
  year = {2026},
  eprint = {2611.19247},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19247},
}
Figure