SceneAgent: A Multi-Agent Framework for Portable Game Scene Generation

Xingyi Li, Jiahao Yang, Yiqin Zhu, Jingwen Ye, Yang Sun, Weidong ZhangarXiv 2026 · arXiv

TL;DR

SceneAgent is a multi-agent framework that makes 3D scene generation portable across different game engines by decoupling high-level design logic from engine-specific implementation. By using a canonical world state and a library of reusable spatial and semantic "skills," it outperforms existing baselines in scene completeness and alignment while allowing the same construction process to work in entirely different backends.

What Problem Does This Paper Address?

Generating a 3D scene with an LLM usually involves the model writing code or calling tools specific to a single environment (like Blender or Unreal Engine). This creates a "lock-in" effect: the planning logic, the coordinate systems, and the learned experience are all tied to one backend. If a developer wants to move their scene generator to a new engine, they often have to start from scratch.

The authors identify this as a lack of construction portability. Beyond just exporting a final file (like a USD or FBX), we need a way to reuse the process of building—the design decisions, spatial rules, and subtask coordination—regardless of whether the final output is for a mobile game engine or a high-end desktop one.

Methods

SceneAgent organizes the generation process around three core mechanisms:

  1. Canonical SceneContext: Instead of agents talking directly to a game engine, they interact with a shared, engine-neutral "world state." This context tracks object transforms, bounding boxes, and occupancy in a standardized coordinate system.
  2. Multi-Agent Protocol: The system uses a hierarchy of agents. An Orchestrator manages the flow; a Planner breaks the user prompt into a queue of semantic subtasks (e.g., "ground," then "buildings," then "vegetation"); and fresh Builder sessions are launched for each subtask. Crucially, Builders only see a compact report of previous work and the current SceneContext, preventing the "long-context" confusion that plagues monolithic agents.
  3. Factored Skill Library: The system distills successful scene builds into reusable, asset-independent skills. These are split into Layout skills (spatial organization, like "place buildings along a road") and Content skills (semantic roles and styles, like "medieval village theme").
Figure 1. SCENEAGENT separates within-run construction from cross-run experience reuse. Fresh Builders coordinate through canonical scene state. Backend adapters realize that state in the target engine. The optional human-gated update path retains approved traces as composable Layout and Content skills.

When moving to a new engine, only the Backend Adapter needs to change. This adapter handles the translation of canonical coordinates and resolves generic asset requests into specific objects from the new engine's library.

Datasets

The authors introduce GameSceneBench, a new benchmark for text-to-3D scene generation.

  • Composition: 69 prompts (56 outdoor, 13 indoor).
  • Diversity: Covers natural landscapes, urban infrastructure, historical settings, and stylized environments.
  • Evaluation: Unlike simple reconstruction metrics, it focuses on alignment, spatial plausibility, and completeness using both human judges and Vision-Language Models (VLMs).

Key Insights and Ideas

  • Structured Handoffs vs. Full History: One of the biggest "aha" moments is that agents don't need the full tool-call transcript of everything built before them. By querying a persistent SceneContext, a "Prop Builder" can respect the space occupied by a "Building Builder" without needing to read the building agent's entire thought process.
  • Combinatorial Skill Reuse: By factoring experience into Layout and Content, the system can mix and match. You can take the layout of a "night market" and realize it with "cyberpunk" content or "ancient Chinese" content without needing new training data for every specific combination.
Example of how Layout and Content skills can be recombined to create different scene variations.
  • The Portability Boundary: The paper clearly defines what is "shared" (planning, skills, state) versus what is "local" (asset catalogs, coordinate conversion). This provides a roadmap for engineers looking to build engine-agnostic AI tools.

Key Results

  • Human Preference: In a blinded study, SceneAgent was preferred over five major baselines (including Holodeck and AutoUE) with scores ranging from 78.2% to 89.4%.
  • VLM Evaluation: GPT-4o-based evaluations favored SceneAgent across all dimensions: prompt alignment, spatial plausibility, and scene completeness.
  • Cross-Engine Success: The authors successfully deployed SceneAgent in a second, different game engine ("Engine B") while keeping the skill library and agent protocol identical, proving the portability claim.
Figure 3. Representative generations in Engine B.

Limitations and Future Work

  • Runtime Latency: The multi-agent coordination comes at a cost. SceneAgent takes an average of 203 seconds per scene, which is significantly slower than monolithic baselines (some of which take under 20 seconds).
  • Human-in-the-Loop: The "Quality Gate" for adding new skills to the library currently requires human approval to ensure high-quality experience distillation.
  • Physicality: While the system uses "Geometry Gates" to check for collisions, it does not yet establish full physical stability, navigability, or gameplay-specific logic (like AI pathfinding).

Future work suggests moving toward more efficient orchestration and incorporating geometry-aware refinement to further reduce the need for human oversight.

Summary updated

Abstract

Moving a scene generator to a new game engine requires more than exporting its final output: planning, construction tools, and stored experience may also depend on the original backend. We present SceneAgent, a multi-agent framework for reusing the construction process and its experience across backends. A canonical SceneContext maintains the evolving world, while backend adapters isolate coordinate conventions, asset bindings, and export. Structured handoffs let fresh Builder sessions query this persistent state without inheriting the full construction transcript. Across generations, human-approved traces are distilled into asset-ID-free Layout skills and semantic Content skills that can be retrieved and recombined. We introduce GameSceneBench, comprising 69 indoor and outdoor prompts. Under a common asset and evaluation protocol, SceneAgent obtains human tie-adjusted preference scores of 78.2-89.4% against five baselines. Ablations show that reusable skills and the multi-agent protocol primarily improve prompt alignment and scene completeness, with additional runtime. Deployment in a second engine preserves the shared construction protocol and skill library while replacing backend integration components. Together, these results support separating persistent scene state, task-local reasoning, and reusable experience for portable scene construction.
BibTeX
@misc{li2026sceneagent,
  author = {Xingyi Li and Jiahao Yang and Yiqin Zhu and Jingwen Ye and Yang Sun and Weidong Zhang},
  title = {{SceneAgent: A Multi-Agent Framework for Portable Game Scene Generation}},
  year = {2026},
  eprint = {2611.17821},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17821},
}
Figure