Automated Environment Synthesis with Large Language Models

50 concepts36 papers1 proposal

To train robust embodied agents, autonomous robots, and complex reasoning models, machine learning relies heavily on simulated environments. However, building these simulations creates a severe development bottleneck. Traditional simulation design requires human engineers to manually write code, place 3D assets, and define reward functions. While traditional Procedural Content Generation (PCG) can randomize environments using hardcoded rules or noise functions, it lacks semantic awareness; a standard PCG algorithm can randomize the coordinates of a knife and a microwave, but it does not inherently understand that a knife belongs on a cutting board and a microwave belongs on a counter.

Automated environment synthesis solves this bottleneck by using Large Language Models (LLMs) as automated simulation engineers. This paradigm works because LLMs combine two core capabilities: a deep, internet-scale understanding of physical and social world statistics, and the ability to generate structured, executable code. By mapping natural language descriptions to simulation parameters, logic, and assets, LLMs allow systems to programmatically self-generate an infinite stream of diverse, semantically coherent training scenarios.

From Natural Language to Spatial Layouts

The first layer of environment synthesis is translating an abstract textual concept (such as "a functional kitchen" or "a cluttered living room") into a valid 3D spatial configuration. LLMs achieve this by acting as semantic orchestrators that interface with 3D asset databases and physics engines.

In RoboGen, Yufei Wang et al. demonstrate a fully automated pipeline where an LLM proposes a task, queries an asset repository to retrieve relevant 3D objects, and programmatically arranges them based on physical plausibility. Similarly, Lirui Wang et al. in GenSim and Pushkal Katara et al. in Gen2Sim utilize LLMs to write the precise Python scripts needed to instantiate assets and configure scenes within robotic simulators.

[Natural Language Prompt] 
       │
       ▼
┌──────────────┐
│  Reviewer/   │ ◄─── Contextual & Ecological Statistics
│  Designer    │      (e.g., RoboCasa, DivScene)
│    LLM       │
└──────┬───────┘
       │ Generates Code / Scene Graphs
       ▼
┌──────────────┐
│ Intermediate │ ───► Task-oriented Scene Graphs (GENMANIP)
│ Structure    │ ───► Hybrid Procedural Layouts (LEGENT)
└──────┬───────┘
       │ Instantiates
       ▼
┌──────────────┐
│ 3D Simulator │ ───► Executable Training Environment
└──────────────┘

To prevent the LLM from generating physically impossible or chaotic layouts, modern frameworks often pair the language model with structured intermediate representations or procedural constraints:

  • Task-oriented Scene Graphs (ToSG): In GENMANIP, Ning Gao et al. introduce a structured graph representation that forces the LLM to map high-level instructions to explicit, logically consistent object placements before rendering the scene.
  • Hybrid Procedural Generation: In LEGENT, Zhili Cheng et al. combine the mathematical guarantees of procedural algorithms with the high-level guidance of LLMs, ensuring that thousands of generated spatial layouts remain physically valid.
  • Ecological Modeling: The authors of RoboCasa leverage LLMs to capture the realistic, multi-stage statistics of human households, while Hongming Zhang et al. scale this approach in DivScene to synthesize thousands of architectural descriptions across dozens of distinct scene categories.

This capability can even operate interactively. As shown in SimWorld by Xiaokang Ye et al., language-driven procedural generation allows users or agent loops to dynamically modify 3D environments on the fly using natural language commands.

Synthesizing Rules, Rewards, and Verifiers

An environment is not merely a static 3D visual layout; it is defined by its underlying mechanics, rules, and success criteria. LLMs are uniquely capable of generating the logical code that governs these dynamics.

For text-based environments, the authors of STARLING (Shreyas Basavatia et al.) demonstrate that LLMs can act as procedural content generators by translating abstract task descriptions into structured game logic, room connectivity matrices, and reward systems. This code-generation capability extends down to low-level systems: in Octax, Waris Radji et al. use LLMs to compile functional arcade environments directly into low-level assembly code for constrained virtual machines.

When environments require tight coupling between physical layouts and abstract rules, LLMs can co-design both simultaneously. As explored in Generating Games via LLMs by Chengpeng Hu et al., providing an LLM with a formal language context allows it to output functional game mechanics alongside matching level designs without breaking logical consistency.

                  ┌─────────────────────────────┐
                  │        Designer LLM         │
                  └──────────────┬──────────────┘
                                 │
         ┌───────────────────────┴───────────────────────┐
         ▼                                               ▼
┌──────────────────────────────┐               ┌──────────────────────────────┐
│        Instance Code         │               │        Verifier Code         │
│  (Generates unique task)     │               │  (Evaluates agent success)   │
└──────────────────────────────┘               └──────────────────────────────┘

For training reasoning models via reinforcement learning, the synthesized environment must provide an infallible ground-truth evaluation signal. Andre Wang He et al. solve this in ReSyn by instructing the LLM to synthesize both the task instance code and its corresponding programmatic verifier. By generating the environment and the evaluator in tandem, the pipeline creates self-contained, verifiable training loops that completely bypass the need for human annotation or pre-computed answer keys.

Contextual Framing and World Modeling

Beyond physical simulations, LLMs can leverage their embedded knowledge of human behavior and social norms to synthesize abstract, narrative-driven environments.

In Framing the Game, Isaac Robinson et al. construct a generative pipeline that procedurally synthesizes thousands of narrative vignettes for game-theoretic scenarios. By systematically altering contextual variables (such as framing an opponent as an "ally" versus an "enemy"), they create synthetic social environments to evaluate how context shapes decision-making.

Similarly, the authors of Did You Hear That? (Ksheeraja Raghavan et al.) employ LLMs as "world models" to simulate complex physical and social scenarios. The model generates structured scene descriptions and identifies contextually anomalous events, automating the creation of targeted benchmark data for anomaly detection.

Open-Ended Evolution and Dynamic Curricula

The ultimate power of automated environment synthesis lies in its ability to create adaptive, open-ended training loops. Because the generation process is programmatic and driven by text, the environment itself can evolve alongside the agent's capabilities.

┌────────────────────────────────────────────────────────┐
│                                                        │
│                  ┌────────────────┐                    │
│                  │  Designer LLM  │                    │
│                  └───────┬────────┘                    │
│                          │ Proposes Parameters         │
│                          ▼                             │
│               ┌────────────────────┐                   │
│               │ Procedural Task Gen│                   │
│               └──────────┬─────────┘                   │
│                          │ Instantiates                │
│                          ▼                             │
│               ┌────────────────────┐                   │
│               │ Target Environment │                   │
│               └──────────┬─────────┘                   │
│                          │ Evaluates Performance       │
│                          ▼                             │
│               ┌────────────────────┐                   │
│               │    Agent Policy    │                   │
│               └────────────────────┘                   │
│                                                        │
└────────────────────────────────────────────────────────┘

Instead of testing models on static benchmarks that suffer from rapid performance saturation, frameworks can optimize environment parameters dynamically. In Automating Benchmark Design, Amanda Dsouza et al. introduce BeTaL (Benchmark Tuning with an LLM-in-the-loop), which uses a reasoning model to iteratively adjust procedural parameters (such as grid sizes or operation counts) until a specific target difficulty is achieved.

For open-ended learning, Fuma Aki et al. introduce LLM-POET, replacing traditional, hand-engineered environment mutation operators with an LLM. The language model mutates environment descriptions based on historical agent performance, evolving a progressively complex curriculum of tasks.

By automating domain randomization and asset scaling—as seen in zero-shot Sim2Real frameworks like DexScale by Guiliang Liu et al.—automated environment synthesis shifts the primary bottleneck of AI development away from human data curation and toward scalable compute. As noted in a comprehensive survey by Roberto Gallotta et al. in Large Language Models and Games, while the field is currently transitioning through a fragmented exploratory phase, the convergence of world knowledge and code generation points toward a future where training simulations are entirely self-generating, adaptive, and open-ended.

Key Foundational Work

Large Language Models and Video Games: A Preliminary Scoping Review (2024) — contributed 5 concept elements to this direction.
ComplexWorld: A Large Language Model-based Interactive Fiction Learning Environment for Text-based Reinforcement Learning Agents (2023) — contributed 3 concept elements to this direction.
Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024 (2024) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_automated_environment_synthesis_1d7976e8,
  title = {automated environment synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-environment-synthesis/rdr_1d7976e8b3524aef8bd50cb07947a33e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_automated_environment_synthesis_1d7976e8_v1,
  title = {automated environment synthesis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/automated-environment-synthesis/rdr_1d7976e8b3524aef8bd50cb07947a33e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake