Abstract
Training robotic policies directly in the real world is expensive and unscalable. Although generative simulation enables large-scale data synthesis, current approaches often fail to generate logically coherent long-horizon tasks and struggle with dynamic physical uncertainties due to open-loop execution. To address these challenges, we propose Affordance-Graphed Task Worlds (AGT-World), a unified framework that autonomously constructs interactive simulated environments and corresponding robot task policies based on real-world observations. Unlike methods relying on random proposals or static replication, AGT-World formalizes the task space as a structured graph, enabling the precise, hierarchical decomposition of complex goals into theoretically grounded atomic primitives. Furthermore, we introduce a Self-Evolution mechanism with hybrid feedback to autonomously refine policies, combining Vision-Language Model reasoning and geometric verification. Extensive experiments demonstrate that our method significantly outperforms in success rates and generalization, achieving a self-improving cycle of proposal, execution, and correction for scalable robot learning.
Introduction
Training robots in physical environments is notoriously slow, expensive, and hazardous. While the "Sim-to-Real" paradigm—training in simulation for transfer to reality—is the standard solution, it faces two critical hurdles. First, the Reality Gap: generative video models often produce visually plausible but physically impossible sequences, such as objects floating or clipping through surfaces. Second, Long-Horizon Failure: most automated systems rely on open-loop execution, where minor errors in early steps (e.g., a slight collision with a door frame) compound until the entire task fails because the robot cannot perceive or correct its plan.
Affordance-Graphed Task Worlds (AGT-World) bridges this "Semantic-Physical Gap" by creating a system that understands both the visual layout of a scene and the functional affordances of its objects. It transforms real-world observations into interactive, physically grounded simulation environments for autonomous robot training.
The AGT-World Framework
The AGT-World pipeline consists of three primary stages designed to move from static images to refined robotic policies.
1. Scene Reconstruction
Rather than purely "imagining" a scene, AGT-World takes a real-world photo and uses a hierarchical search to match objects to interactive assets from the BEHAVIOR-1K dataset. These assets are instantiated within the OmniGibson physics simulator, preserving the semantic and spatial layout of the original environment.
2. Graph-Based Task Modeling
The framework formalizes the task space as a 3D Semantic-Action Tensor, treating task generation as a path-finding problem on a structured graph. High-level user goals (e.g., "put the glass in the fridge") are decomposed into atomic primitives like GRASP or OPEN. A key innovation is the use of Inter-Task Edges (Action Transfers), which ensure logical transitions between subtasks by verifying that objects are in the correct state to accept subsequent actions.
3. Self-Evolution Mechanism
The core of AGT-World is its closed-loop "Self-Evolution" cycle, which replaces traditional open-loop planning with a self-improving feedback loop:
- The Inspector (VLM): After an execution attempt, a Vision-Language Model (VLM) reviews multi-view video frames to perform motion verification and anomaly detection.
- The Supervisor (VLM): Using the Inspector's critique and execution history, the Supervisor adjusts the action sequence or geometric parameters (e.g., "move the base 0.3m left") for the next iteration.
Experimental Evaluation
The framework was evaluated on a new benchmark of 102 unique autonomous scenarios derived from 34 diverse real-world scenes (kitchens, bedrooms, etc.). AGT-World was compared against established baselines such as RoboGen, Behavior-100, and ManiSkill2.
Key Results
- High Success Rate: The system achieved a 71.6% overall success rate across the generated tasks.
- Semantic Fidelity: AGT-World outperformed baselines in S-BERT similarity (0.376), indicating that generated tasks remained faithful to user intent without hallucinating irrelevant actions.
- Structural Consistency: It achieved a Self-BLEU score of 0.806, indicating a highly standardized and executable program format.
- Rescue Capability: The Self-Evolution mechanism proved highly effective, with an Evolution Success Rate (ESR) of 62.5% when using models like Gemini-3-Flash to rescue initially failed episodes.
Insights and Future Directions
A central insight of this work is that successful task transfer depends on Functional Switching—ensuring an object is in a valid state for the next function—rather than just kinematic continuity. By using VLMs as "process supervisors" rather than just high-level planners, AGT-World can resolve physical uncertainties that are impossible to predict in advance.
Current limitations include monocular scale ambiguity, where the system struggles to determine absolute object sizes from a single 2D image. Future work will explore integrating Reinforcement Learning (RL) agents for low-level execution and developing more robust sim-to-real pathways for physical hardware testing.
Summary updated