Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph

Xiang Liu, Sen Cui, Guocai Yao, Zhong Cao, Jingheng Ma, Min Zhang, Changshui Zhang

Here is a concise, engaging summary of the Scene2Demo research paper.

TL;DR

Scene2Demo is an automated pipeline that transforms a single real-world RGB image and a text query into a fully interactive simulation environment populated with executable robot training data. By combining a structured object-action graph with a closed-loop visual feedback mechanism, the system can automatically detect its own execution failures and refine its action plans, producing high-quality datasets ready for downstream imitation learning.

What Problem Does This Paper Address?

To train capable embodied AI agents, researchers need massive amounts of interaction data that are physically valid, executable, and diverse. Collecting this data in the real world is notoriously slow and expensive. While generative AI and physics simulators offer a path toward automated data generation, existing methods fall short:

  1. Video generation models often violate basic physical laws.
  2. Unconstrained Large Language Models (LLMs) used for task planning frequently generate inconsistent or unexecutable subtask structures.
  3. Open-loop execution in simulators fails when asset sizes, navigation offsets, or reconstruction errors cause the robot to miss an object or collide with the environment.

Scene2Demo bridges this gap by creating a reliable, self-correcting data generation engine that turns static images into rich, multi-view robot demonstration datasets.

Methods

The Scene2Demo framework operates in two main stages: Graph-Based Task Generation and Agentic Self-Evolution.

Figure 1. An introduction of our pipeline.

1. Graph-Based Task Generation

First, the system reconstructs a 3D interactive simulation substrate from a single RGB image using an existing pipeline (following Digital Cousins). On top of this substrate, Scene2Demo uses a structured multi-module workflow governed by an object-action graph.

Instead of letting an LLM freely write code, the task is strictly decomposed into symbolic definitions (using the Behavior Domain Definition Language, or BDDL). The pipeline automatically adjusts object sizes to fit the robot's gripper constraints and maps the subtasks to an initial sequence of primitives (e.g., APPROACH, GRASP, MOVE_BASE) from a predefined library.

2. Self-Evolution via Agentic Feedback

If the initial plan fails during simulation, the framework enters a self-evolution loop driven by two specialized vision-language model (VLM) agents:

  • Robotic Safety Inspector: Analyzes multi-view video frames locally to provide step-wise critiques (verifying motion, state transitions, and anomalies).
  • Robotic Task Supervisor: Aggregates these critiques with the broader task context and history to revise the action flow, adjusting parameters (like distances or angles) or modifying the action sequence.
Figure 3. Graph-Based Task Generation with Self-Evolution. For details, see Appendix A.2,A.3.

Datasets

The authors evaluated Scene2Demo using:

  • 102 automatically generated primitive scene-task pairs derived from 34 unique real-world reference images (covering 16 kitchens, 10 bedrooms, and 8 other household layouts).
  • BEHAVIOR-1K asset dataset to match virtual objects (like fridges, cups, and apples) to the real-world scenes.
  • Four representative long-horizon tasks used for benchmarking: putting a glass/apple into a refrigerator, putting an apple into a bowl, and placing a cup on a table.

Key Insights and Ideas

  • The Power of Constraints: Relying on unconstrained LLMs for task decomposition leads to highly erratic plans. Grounding the decomposition in a formal object-action graph ensures that the same user query consistently yields the same logical subtask units across repeated runs.
  • Multi-View Context is Mandatory: Ablation studies showed that using a single camera view drastically degrades performance. Combining a Global View (for overall task logic) with a Head View (for robot joint states) is essential for diagnosing subtle geometric failures, such as a robotic arm link grazing a refrigerator door frame.
  • Error Accumulation in Long-Horizon Tasks: Self-evolution is most critical in the later stages of long-horizon tasks. While early subtasks like opening a door or picking up an object usually succeed on the first try, later subtasks (like precise placing) frequently require multiple iterations to correct accumulated navigation and placement drift.
Figure 4. Detailed Action Display of Self-Evolution. We take the task T1 (Glass→Refrigerator) as an example. The task is always decomposed into four subtasks: “open refrigerator”, “pick up glass”, “put glass into refrigerator”, and “close refrigerator”.

Key Results

  • High Primitive Success: Across the 102 automated primitive tasks, Scene2Demo achieved a 71.6% execution success rate without any human intervention.
  • Superior Long-Horizon Performance: On complex, multi-step tasks, the self-evolution mechanism dramatically improved success rates, outperforming automated baselines like RoboGen and GenSim2.
  • Downstream Policy Viability: To prove the data's utility, the authors trained standard behavior cloning (BC) policies on 100 autonomously generated demonstrations. The trained policies achieved a 96.0% success rate on opening a refrigerator and a 92.0% success rate on picking up a glass, confirming that the generated data is diverse and clean enough to train robust control policies.

Limitations and Future Work

While promising, the authors note several limitations:

  • Primitive Action Space: The current library is restricted to basic actions (Open/Close, Pick/Place, Navigation) and cannot handle fine-grained, dexterous manipulation like folding clothes.
  • Reconstruction Dependencies: Failures can cascade from the initial 3D scene reconstruction stage if interactive assets are misaligned or missing functional parts.
  • VLM Bottlenecks: The self-evolution loop is entirely dependent on the visual reasoning capabilities of the underlying VLM, which can still struggle with precise spatial calculations.

Future work will focus on integrating learned low-level controllers to expand the action library and building a "skill documentation library" to store and reuse successful correction strategies.

Summary updated

Abstract

We present Scene2Demo, a self-evolving framework for offline embodied data generation. Given a single real-world RGB image and a user query, Scene2Demo constructs an interactive simulated scene and generates executable task configurations, multi-view execution videos, and offline robot-learning datasets. Scene2Demo uses a structured multi-module workflow via an object-action graph, representing task generation through object-centric configurations and action transitions. Failed or incomplete executions are further refined by feedback agents that inspect visual rollouts and revise action flows through sequence modification or parameter adjustment. Across 102 automatically generated primitive scene-task pairs, Scene2Demo achieves a 71.6\% execution success rate; on four representative long-horizon tasks, self-evolution improves both task success and subtask-level execution quality over primitive-only execution, and comparisons with RoboGen and GenSim2 show stronger task planning and execution performance under automated data-generation settings. Finally, behavior cloning policies achieve 96.0\% and 92.0\% success on two representative tasks, validating that the generated data can support downstream policy learning. Our project page is available at https://scene2demo-anon.github.io/ .
BibTeX
@misc{liu2026scene,
  author = {Xiang Liu and Sen Cui and Guocai Yao and Zhong Cao and Jingheng Ma and Min Zhang and Changshui Zhang},
  title = {{Scene2Demo: Self-Evolving Embodied Data Generation via Object-Action Graph}},
  year = {2026},
  eprint = {2602.12065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.12065},
}
Figure