ContextRenderer: Dynamic-Context Video Generation for Interactive Game Worlds

Runze Wang, Fei Zha, Bing He, Ke Xu, Zhongyuan Wu, Yuxuan Xie, Wenzheng Wu, Zhongyuan Liu, Youcheng Cai, Ligang LiuarXiv 2026 · arXiv

TL;DR

ContextRenderer is a novel autoregressive video diffusion framework that acts as a "generative renderer" for interactive game worlds. By combining multiscale history compression with a cross-proxy alignment strategy, it enables minute-long, consistent gameplay sessions where players can dynamically change characters, prompts, and visual styles on the fly.

What Problem Does This Paper Address?

Building interactive "world models" is the current frontier of AI in gaming. Recent approaches separate the world engine (which handles logic, physics, and state) from the visual generator (which turns that state into pixels).

However, existing generative renderers face two major hurdles:

  1. Visual Drift in Long Sessions: As gameplay continues, models struggle to maintain visual continuity. If a new character enters or a prompt changes, the model often "forgets" the previous state or introduces jarring artifacts.
  2. Proxy Rigidity: Most models are trained on a single type of input (e.g., just depth maps). In a real game development pipeline, you might want to render from low-poly meshes, semantic maps, or simple "blockouts."

ContextRenderer tackles these by introducing a framework that is both dynamically conditionable and invariant to the specific type of "proxy" representation used by the game engine.

ContextRenderer generates successive video chunks from refreshed world context and visual history.

Methods

The authors introduce three core technical innovations within an autoregressive video diffusion pipeline:

  • Dynamic-Context Learning (DCL): This training scheme allows the model to handle "refreshed" world context at every video chunk boundary. It supervises the model to follow new interaction prompts and subject references (like a new character image) while preserving the appearance of unaffected background elements.
  • Recurrent History Compression (RHC): To stay within a fixed "token budget" for the Transformer, RHC uses a multiscale approach. It keeps a high-resolution "anchor" of the very first frame and uses finer patches for recent frames, while older history is compressed into coarser tokens. This provides both immediate motion cues and long-term context.
  • Cross-Representation Flow Alignment (CRFA): This is the "secret sauce" for proxy generalization. During training, the model learns to map different proxies (depth, voxels, low-poly) to a shared high-fidelity reconstruction. This teaches the model to focus on the underlying layout and motion rather than the specific style of the input proxy.
The ContextRenderer architecture, highlighting the RHC history compression and the CRFA training flow.

Datasets

The researchers curated WorldPrism, a massive interaction-centric dataset:

  • Scale: 20,000+ clips.
  • Diversity: 150+ visual styles, 11 proxy formats (depth, semantic, low-poly, etc.), and 100+ different game worlds.
  • Source: Data was pulled from agent-built worlds, AAA gameplay, and open-source video datasets.

For evaluation, they used GWM-Bench (Game World Model Benchmark) and Solar-Bench to test reconstruction fidelity and visual variation.

Key Insights and Ideas

  • The "First-Frame Anchor" is Essential: One of the simplest but most effective ideas in RHC is keeping the very first frame of a session as a permanent, high-fidelity reference. This acts as a visual "north star," preventing the identity of characters or environments from drifting over minutes of gameplay.
  • Shared Latent Targets: By forcing different proxies to align with a single high-fidelity reconstruction (CRFA), the model becomes a "universal translator" for game engine outputs. You can swap a voxel proxy for a semantic one mid-session, and the output remains coherent.
  • Chunk-Level Supervision: Instead of training on entire long videos, supervising at the "chunk" level with updated prompts mimics the actual loop of an interactive game engine.
Demonstration of "any interaction, any proxy, any look" across different rendering styles.

Key Results

ContextRenderer was tested across various model scales (5B, 14B, and 33B parameters).

  • Quantitative: The 33B model achieved a significantly lower LPIPS (perceptual loss) of 0.222 on GWM-Bench, outperforming baselines like SolarRenderer (0.254) and EdenLive (0.257).
  • Temporal Consistency: It maintained high "Adjacent-Frame Similarity" (0.983), indicating very low flicker.
  • Qualitative: The model successfully rendered minute-long interactive sessions (Figure 5) where the player drives, exits a car, and engages in combat, with the model correctly "recalling" the car's appearance when the player returns to it after several interaction chunks.
A minute-long interactive session showing consistent character and vehicle identity across multiple state changes.

Limitations and Future Work

  • Inference Speed: While the paper demonstrates high quality, real-time interaction (e.g., 30+ FPS) still requires further optimization through quantization or distillation.
  • Memory Horizon: The bounded history means the model might eventually "forget" details from very distant parts of a massive open world. The authors suggest explicit retrieval mechanisms (like a long-term memory bank) as a future direction.
  • Data Coverage: While WorldPrism is large, the authors note that even broader training data is needed to cover more niche art styles and complex physical interactions.

Summary updated

Abstract

Interactive game world models separate executable world evolution from visual generation, using video models to render engine-provided scene representations into detailed observations. During sustained gameplay, however, existing generative renderers struggle to follow changing conditions while preserving visual continuity across diverse proxies and target styles. We therefore introduce ContextRenderer, an autoregressive video diffusion framework designed for both dynamic conditioning and cross-proxy generalization. It couples Dynamic-Context Learning (DCL) for chunk-level interaction supervision with Recurrent History Compression (RHC) to follow condition updates while preserving visual continuity within a bounded token budget. Across proxy representations, Cross-Representation Flow Alignment (CRFA) connects proxy reconstruction with appearance and interaction variation through a shared high-fidelity reconstruction. WorldPrism, our interaction-centric dataset constructed from agent-built worlds, AAA gameplay, and open-source videos, supplies the corresponding renderings and chunk-level annotations for these learning tasks. Experiments on GWM-Bench and Solar-Bench show improved reconstruction and visual quality, while qualitative results demonstrate minute-long interactive rendering under changing prompts and subject references. Project page: https://rainzor.github.io/contextrenderer/
BibTeX
@misc{wang2026contextrenderer,
  author = {Runze Wang and Fei Zha and Bing He and Ke Xu and Zhongyuan Wu and Yuxuan Xie and Wenzheng Wu and Zhongyuan Liu and Youcheng Cai and Ligang Liu},
  title = {{ContextRenderer: Dynamic-Context Video Generation for Interactive Game Worlds}},
  year = {2026},
  eprint = {2611.25838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25838},
}
Figure