Robots that navigate whole buildings cannot keep a single dense map of everything they have seen. Memory and planning cost grow with the area covered, so practical systems break space into a graph of places (rooms, corridors, stairwells), each with its own small local map. Simulators used to train these robots work the other way. Language-model scene generators such as Holodeck write out one flat set of global coordinates. They stay realistic for a single floor, but they break down on multi-story buildings and on large repeated structures. Holodeck reports exactly these failures.
We propose to generate buildings the way scalable robots map them. A large language model (LLM) first writes a place graph: nodes are rooms or corridor segments, and edges are doors, openings or stairs. It then fills in each node separately, inside a bounded local window, using only that node's neighborhood as context. Repeated wings and floors are written as program loops rather than copied coordinates. The result is a scene together with a ground-truth topological map and per-node metric patches. This paired output lets us test something no current benchmark supports: how navigation agents that use bounded-memory maps behave as buildings grow, repeat, and change, in worlds that nobody had to author by hand.
Core Claim
Using the place graph as the unit of LLM context lets language-guided generation reach building scale, meaning multiple floors and dozens of rooms with repeated sections, without the global-coordinate hallucinations seen in flat generation. Because the graph is emitted alongside the geometry, the generated worlds come with exact topological supervision. Training topological navigators on them should transfer better to large real scans than training on procedural or single-floor data. The novelty is not "an LLM lays out rooms," which Holodeck already does. It has three parts: a spatial graph serves as the context factorizer, the graph is a first-class output used as map ground truth, and scale, repetition and layout change can be set deliberately as experimental knobs.
Research Program
The method borrows from three places.
The first is FactorSim. FactorSim showed that restricting what the LLM sees at each step of simulation-code generation cuts hallucinated APIs and raises task completion on its benchmark from 8% to 33%. We replace its model-view-controller split with a spatial split. The working hypothesis is that a building's place graph is the natural factorization for 3D layout, as a game's state variables are for game code.
The second is the hybrid map tradition of Integrating Grid-Based and Topological Maps and NavTopo. These systems plan globally over a sparse graph and locally over small grids. That structure fixes our output format: each node owns a metric patch of bounded size, anchored at its door edges.
The third is The Scene Language, which represents scenes as executable programs with loops. Following it, the LLM expresses repetition ("twelve identical hotel rooms on each of four floors") as a loop over a room template. This is the case Holodeck cannot handle when coordinates are written out one by one.
Generation runs in three stages. In stage one, the LLM receives a building brief such as a clinic, an office tower or a hotel. It writes a graph program listing node types, rough areas, adjacency, door widths and floor membership. A numerical solver then turns these into non-overlapping footprints, keeping Holodeck's useful choice of letting a solver handle exact coordinates. In stage two, the LLM sees one node and its immediate neighbors at a time. It selects and places assets inside a local frame, again through relational constraints and a solver, using Holodeck's annotated Objaverse library and ProcTHOR's interactive physics assets. In stage three, an automatic checker computes a navigation mesh, which is the walkable surface. It verifies three things: every graph edge is actually traversable, no node exceeds its window, and the graph derived from the mesh matches the planned graph. If an edge fails, the checker sends back a local repair request for the two nodes involved, as in the self-correcting loop of SLD. It does not regenerate the whole building.
The generator also exposes three parameters that target known weak points of topological mapping. The first is building size, measured as node count and floor count. The second is perceptual aliasing, the case where distinct places look the same, which we vary as the fraction of nodes instantiated from shared templates. The third is layout change: closing doors or moving furniture between sessions, which is the lifelong-update problem the mapping literature leaves open.
Baselines
The naive reference point is the strongest existing generator run without modification. Holodeck is prompted with the same building brief and asked to produce the full multi-floor layout in one pass. It gets the same LLM, asset library, solver and per-building token budget as our method, with best-of-k sampling up to that budget. Holodeck reports 64.4% human preference over procedural scenes and raised NoveltyTHOR music-room navigation success from about 6% to over 31%. Lacuna has no reported result for its validity on multi-floor buildings, so we measure that ourselves. To beat it, our method must achieve a clearly higher rate of fully connected, collision-free, in-bounds buildings at 20 or more nodes and at two or more floors, without losing human preference on single floors.
The procedural competitor is ProcTHOR's boundary-cutting generator. We extend it to multiple floors by stacking floors connected through stair nodes and give it the same asset set. Its scale is real: 10,000 houses per hour. But HSSD-200 reports that agents trained on 10,000 ProcTHOR scenes reach 12.5% ObjectNav success on HM3DSem, against 19.2% for 122 human-authored HSSD scenes. That gap is the margin LLM structure would need to close.
The human-authored reference is training directly on HSSD and on the multi-floor real scans of HM3D. Scenes from these datasets are held out by building for evaluation.
The key ablation keeps everything in our pipeline but removes graph factorization. The LLM still writes a graph program, but then places all assets in one global context. A second ablation removes program loops and enumerates repeated rooms explicitly. These two ablations separate the effect of context factorization from the effect of compact repetition.
Why not just…?
Why not just prompt Holodeck hierarchically, floorplan first and then rooms? This is the closest threat. Holodeck already plans the floorplan before furnishing rooms. The differences are scope and output. Holodeck's floorplan step is single-floor and coordinate-level. It does not give the LLM bounded local frames, does not encode repetition as loops, and returns no verified place graph. The "no factorization" ablation and the multi-floor Holodeck baseline together measure how much of the gain comes from the graph. If a carefully prompted Holodeck matches us on validity, the remaining contribution is the paired map ground truth and the change knobs, and we would report it as that.
Why not just use procedural floorplans, since realism may not matter for mapping? This is plausible for pure memory and latency studies, where wall geometry dominates. It is weaker for transfer. HSSD-200's 12.5% versus 19.2% result shows that procedural realism limits real-world navigation. Experiment two settles the question directly. If ProcTHOR-style buildings transfer as well as ours to HM3D topological navigation, the LLM is unnecessary for this purpose.
Why not just use HM3D, which already has scans with one to eight floors? HM3D is fixed in size. It cannot vary aliasing or simulate closed doors, and it provides no ground-truth place graph. It is also finite at about 1,000 scans. We use it as the held-out real target, not as the training distribution.
Proposed Experiments
The first experiment tests generation validity at scale. We generate buildings of 5 to 80 nodes and 1 to 6 floors across ten building types. For each configuration we report the fraction that pass the navigation-mesh connectivity and collision checks, out-of-bounds rate (the metric LayoutGPT used, where it fell from 83% to 64%), agreement between planned and derived graphs, and tokens consumed. The decisive check is whether the validity of monolithic generation falls with node count while ours stays roughly flat.
The second experiment tests transfer. We train the learned reachability estimator and local controller of Scaling Local Control separately on our buildings, ProcTHOR-style buildings and HSSD, with matched numbers of trajectories. That work reports 84.6% to 95.7% success on 5 unseen Gibson environments. We evaluate on those same environments and on held-out large multi-floor HM3D buildings, reporting success, path-length-weighted success (SPL), and graph vertex count. In parallel, we evaluate ObjectNav on HM3D using the VoroNav graph navigator, whose reported 42.0% success rate on HM3D is the reference.
The third experiment uses the generator as a scaling testbed. We run NavTopo and a dense RTAB-MAP-style mapper on buildings of increasing size and aliasing. We record map memory, planning latency and false loop closures, which are wrong "I have been here before" matches. This extends NavTopo's single-corridor result of 57 MB versus 352 MB and 6 ms versus 830 ms to controlled building scale. Change sessions with closed doors test whether the mappers repair their graphs correctly. The ground-truth graph makes graph-repair accuracy directly measurable for the first time.
The claim fails if either of two things happens. Monolithic Holodeck may match our validity at 40 or more nodes. Or reachability models trained on our worlds may transfer to HM3D no better than procedural ones. Either outcome would be informative about what LLM structure contributes to simulation.
References
- Holodeck: Language Guided Generation of 3D Embodied AI Environments
- FactorSim: Generative Simulation via Factorized Representation
- Integrating Grid-Based and Topological Maps for Mobile Robot Navigation
- NavTopo: Leveraging Topological Maps for Autonomous Navigation of a Mobile Robot
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings
- 🏘️ ProcTHOR: Large-Scale Embodied AI Using Procedural Generation
- Self-Correcting LLM-Controlled Diffusion Models
- Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation
- Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI
- LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
- Scaling Local Control to Large-Scale Topological Navigation
- VoroNav: Voronoi-based Zero-shot Object Navigation with Large Language Model