Generative Simulation for Scaling AI Agents

This direction is merged into another direction: manual asset creation bottleneck. Its last version stays readable and citable here.

Version 1 of 1101 papers102 conceptsfrozen 2026-10-04

The primary bottleneck in training general-purpose AI agents—whether they are robots navigating a kitchen or LLMs using software tools—is no longer the model architecture, but the availability of diverse, interactive environments. Traditionally, creating a simulation requires human experts to manually code physics, place 3D assets, and define reward functions. This "manual bottleneck" limits agents to a handful of curated tasks, leading to brittle policies that fail in the messy complexity of the real world.

Generative Simulation is a research direction that seeks to automate the creation of these environments. By leveraging foundation models to "hallucinate" executable code, 3D scenes, and interactive logic, researchers are moving toward a paradigm of "environment scaling," where agents can be trained on an infinite stream of synthetic but physically grounded experiences.

The Manual Bottleneck in Simulation

In traditional pipelines, every new task requires a "digital twin." As noted by Seung Wook Kim and colleagues in the development of GameGAN, explicitly coding the rules of a game or a physical environment is labor-intensive and fails to scale. This problem is even more acute in robotics; Katerina Fragkiadaki and the RoboGen team argue that the high cost of manual data collection (like teleoperation) necessitates a "self-generating" loop where models propose their own tasks and environments.

LLMs as Environment Architects

The most direct way to automate simulation is to use Large Language Models (LLMs) to write the simulation code itself. Systems like GenSim and Holodeck (developed by Yue Yang and Luca Weihs) use GPT-4 to translate high-level natural language prompts into structured 3D scenes and reward functions.

The intuition here is that LLMs possess "common-sense" spatial reasoning. If you ask for a "messy living room," an LLM knows that a coffee cup belongs on a table, not the ceiling. By combining this latent knowledge with a library of 3D assets, Holodeck can "prompt" entire training worlds into existence. To ensure these generated worlds are actually useful for training, frameworks like FactorSim use factorized representations to reduce the complexity of the code the LLM must generate, ensuring the resulting simulations are stable enough for reinforcement learning.

Real-to-Sim: Turning Video into Interactive Worlds

While LLMs can dream up new scenes, another branch of generative simulation focuses on converting the real world into digital twins. UrbanVerse and RobotArena $\infty$ (introduced by Yash Jangir and others) use "real-to-sim" pipelines to transform passive data—like YouTube city tours or RGB-D images—into interactive, physics-aware environments.

This approach solves the "fidelity gap." Instead of a human artist trying to replicate the texture of a sidewalk, the GRS system uses Vision-Language Models (VLMs) to understand a real-world image and then automatically generates the corresponding simulation code. This allows robots to practice in a digital replica of the exact environment they will eventually deploy in.

Emulated Sandboxes: Bypassing the Physics Engine

In domains like cybersecurity or tool-use, the "environment" isn't a 3D world but a set of APIs or terminal commands. Building a secure, reproducible sandbox for every possible API is an engineering nightmare.

Researchers like Yangjun Ruan and Jimmy Ba have proposed ToolEmu, which uses an LLM to emulate the environment. When an agent tries to "delete a file" or "transfer money," the emulator LLM predicts what the output of that command would be. This "implicit world model" allows for the stress-testing of agents in high-stakes scenarios—like CTF-Dojo for cybersecurity—without the overhead of building custom infrastructure for every task.

Why It Works: The Power of Environment Scaling

The core insight of this research direction is that environment diversity is a proxy for general intelligence. By scaling the number of environments rather than just the number of parameters, models like AgentScaler (from Alibaba's Tongyi Lab) have shown that a 30B-parameter model can outperform 1T-parameter giants simply because it has "seen" more diverse functional domains.

Generative simulation turns the "manual bottleneck" into a "computational scaling law." As foundation models get better at reasoning and coding, the simulations they generate become more complex and accurate, providing a self-improving flywheel for the next generation of autonomous agents.

Papers in this version

Showing 60 of 101 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generative_simulation_for_scaling_ai_age_5ff75c43,
  title = {Generative Simulation for Scaling AI Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-simulation-for-scaling-ai-agents/rdr_5ff75c4375c24078ab4c476480296b3a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_generative_simulation_for_scaling_ai_age_5ff75c43_v1,
  title = {Generative Simulation for Scaling AI Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generative-simulation-for-scaling-ai-agents/rdr_5ff75c4375c24078ab4c476480296b3a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}