Abstract
Scaling robot learning is currently hindered by the "human bottleneck" in data collection and simulation design. We present RoboGen, a fully automated pipeline that leverages foundation models to self-generate an endless stream of robotic tasks, simulation environments, and training supervisions. By orchestrating Large Language Models (LLMs), Vision-Language Models (VLMs), and physics-based simulators, RoboGen automates the entire "propose-generate-learn" cycle. Our results demonstrate that RoboGen can produce highly diverse tasks—spanning rigid, articulated, and soft-body manipulation—achieving significant semantic variety and high success rates with minimal human intervention.
Introduction
The recent success of Large Language Models (LLMs) is largely attributed to the availability of massive internet-scale datasets. In contrast, robotics faces a severe "data hunger" problem. Physical interaction data is slow, expensive, and potentially dangerous to collect in the real world. While simulation offers a scalable alternative, existing benchmarks rely on human experts to manually design tasks, retrieve 3D assets, and write reward functions. This manual labor prevents simulation data from scaling to the levels required for generalist robots. RoboGen addresses this by automating the simulation director's role, closing the loop between task proposal and policy learning.
The RoboGen Pipeline
RoboGen functions as an automated factory for robotic skills, utilizing a suite of foundation models to manage four distinct stages:
- Task Proposal: GPT-4 samples robots and objects from datasets like PartNetMobility to propose semantically meaningful tasks (e.g., "heat up soup in a microwave").
- Scene Generation: The system populates the environment by retrieving 3D meshes from Objaverse. A VLM (Gemini-Pro) serves as a verification layer, ensuring retrieved assets match the task's semantic requirements. For soft-body tasks, a text-to-image-to-3D pipeline is employed.
- Training Supervision Generation: GPT-4 decomposes high-level tasks into sub-steps and selects the optimal learning tool—Reinforcement Learning (RL), Motion Planning, or Trajectory Optimization—based on the task's physical nature. It also automatically generates the Python code for reward functions.
- Skill Learning: The robot acquires the skill within Genesis, a multi-material differentiable simulator.
Experimental Results
RoboGen was evaluated against established human-crafted benchmarks such as Meta-World and ManiSkill2. The system introduced a new set of 106 automatically generated tasks spanning various manipulation types and legged locomotion.
Diversity and Success Rates
A key metric for RoboGen is its ability to generate diverse content. Using Self-BLEU scores to measure semantic and visual diversity, RoboGen achieved a score of 0.284, significantly outperforming Meta-World (0.322). This indicates that the automated pipeline produces a wider variety of tasks than human designers typically do. Across 69 benchmarked tasks, the system achieved an average success rate of 77.4%.
The Role of Verification
Ablation studies highlighted the necessity of the VLM verification step. Without "size verification" (checking if relative object scales are physically plausible), simulation validity dropped sharply. The VLM acts as a critical sanity check, preventing the system from attempting impossible tasks, such as placing a large object into a smaller container.
Discussion and Future Work
While RoboGen significantly scales task generation, challenges remain. The "sim-to-real" gap persists, requiring further research into domain randomization to transfer these learned skills to physical hardware. Additionally, while LLMs are excellent orchestrators, they still struggle with high-precision spatial reasoning, such as the exact clearances required for lock-and-key tasks. Future iterations of RoboGen will focus on incorporating multi-modal feedback loops, allowing the robot to identify and correct faulty reward functions during the learning process.
Summary updated