TL;DR
SceneAgent is a multi-agent framework that makes 3D scene generation portable across different game engines by decoupling high-level design logic from engine-specific implementation. By using a canonical world state and a library of reusable spatial and semantic "skills," it outperforms existing baselines in scene completeness and alignment while allowing the same construction process to work in entirely different backends.
What Problem Does This Paper Address?
Generating a 3D scene with an LLM usually involves the model writing code or calling tools specific to a single environment (like Blender or Unreal Engine). This creates a "lock-in" effect: the planning logic, the coordinate systems, and the learned experience are all tied to one backend. If a developer wants to move their scene generator to a new engine, they often have to start from scratch.
The authors identify this as a lack of construction portability. Beyond just exporting a final file (like a USD or FBX), we need a way to reuse the process of building—the design decisions, spatial rules, and subtask coordination—regardless of whether the final output is for a mobile game engine or a high-end desktop one.
Methods
SceneAgent organizes the generation process around three core mechanisms:
- Canonical SceneContext: Instead of agents talking directly to a game engine, they interact with a shared, engine-neutral "world state." This context tracks object transforms, bounding boxes, and occupancy in a standardized coordinate system.
- Multi-Agent Protocol: The system uses a hierarchy of agents. An Orchestrator manages the flow; a Planner breaks the user prompt into a queue of semantic subtasks (e.g., "ground," then "buildings," then "vegetation"); and fresh Builder sessions are launched for each subtask. Crucially, Builders only see a compact report of previous work and the current
SceneContext, preventing the "long-context" confusion that plagues monolithic agents. - Factored Skill Library: The system distills successful scene builds into reusable, asset-independent skills. These are split into Layout skills (spatial organization, like "place buildings along a road") and Content skills (semantic roles and styles, like "medieval village theme").
When moving to a new engine, only the Backend Adapter needs to change. This adapter handles the translation of canonical coordinates and resolves generic asset requests into specific objects from the new engine's library.
Datasets
The authors introduce GameSceneBench, a new benchmark for text-to-3D scene generation.
- Composition: 69 prompts (56 outdoor, 13 indoor).
- Diversity: Covers natural landscapes, urban infrastructure, historical settings, and stylized environments.
- Evaluation: Unlike simple reconstruction metrics, it focuses on alignment, spatial plausibility, and completeness using both human judges and Vision-Language Models (VLMs).
Key Insights and Ideas
- Structured Handoffs vs. Full History: One of the biggest "aha" moments is that agents don't need the full tool-call transcript of everything built before them. By querying a persistent
SceneContext, a "Prop Builder" can respect the space occupied by a "Building Builder" without needing to read the building agent's entire thought process. - Combinatorial Skill Reuse: By factoring experience into Layout and Content, the system can mix and match. You can take the layout of a "night market" and realize it with "cyberpunk" content or "ancient Chinese" content without needing new training data for every specific combination.
- The Portability Boundary: The paper clearly defines what is "shared" (planning, skills, state) versus what is "local" (asset catalogs, coordinate conversion). This provides a roadmap for engineers looking to build engine-agnostic AI tools.
Key Results
- Human Preference: In a blinded study, SceneAgent was preferred over five major baselines (including Holodeck and AutoUE) with scores ranging from 78.2% to 89.4%.
- VLM Evaluation: GPT-4o-based evaluations favored SceneAgent across all dimensions: prompt alignment, spatial plausibility, and scene completeness.
- Cross-Engine Success: The authors successfully deployed SceneAgent in a second, different game engine ("Engine B") while keeping the skill library and agent protocol identical, proving the portability claim.
Limitations and Future Work
- Runtime Latency: The multi-agent coordination comes at a cost. SceneAgent takes an average of 203 seconds per scene, which is significantly slower than monolithic baselines (some of which take under 20 seconds).
- Human-in-the-Loop: The "Quality Gate" for adding new skills to the library currently requires human approval to ensure high-quality experience distillation.
- Physicality: While the system uses "Geometry Gates" to check for collisions, it does not yet establish full physical stability, navigability, or gameplay-specific logic (like AI pathfinding).
Future work suggests moving toward more efficient orchestration and incorporating geometry-aware refinement to further reduce the need for human oversight.
Summary updated