Introduction
In fields like Computer Vision and Natural Language Processing, scaling data and compute has led to highly capable "zero-shot" foundation models like GPT-4. Robotics, however, has noticeably lagged behind. The dominant response within the research community has been to scale up physics simulations—creating increasingly complex, parallelized virtual environments to train robots before deploying them in the real world (the Sim2Real approach).
This position paper challenges that prevailing paradigm. It argues that massive physics simulations are neither necessary nor sufficient for achieving general-purpose, human-compliant robot manipulation. Instead, the author advocates for a fundamental shift toward learning from diverse real-world data and passive human videos.
The Fallacy of "Perfect" Simulation
The core argument against scaling simulation rests on what the author terms the "Harder Problem" fallacy. To simulate a task like pouring water perfectly, a system must solve complex fluid dynamics and particle interactions. Paradoxically, this is actually a harder computational problem than the physical manipulation task itself. Humans successfully pour water every day without possessing a perfect internal physics engine; we rely on coarse reasoning, intuition, and continuous sensory feedback.
Furthermore, high success rates in simulation (e.g., achieving 90% accuracy at opening virtual drawers) frequently fail to correlate with real-world success. This "sim-to-real gap" occurs because simulators often lack the right parameters, rather than just needing more of them. Even with techniques like Domain Randomization, a simulator that is fundamentally inaccurate cannot simply "randomize" its way into reflecting reality.
Real-World Desiderata: Zero-Shot Deployment and Human Preferences
The paper evaluates the "Scaling Simulation" approach against two critical requirements for widely usable, "in-the-wild" robots:
- Zero-Shot Deployment: A robot must be able to operate out-of-the-box in a novel environment without requiring extensive fine-tuning. The chronic lack of generalization in robotics is highlighted by the "Green Background" problem—an anecdote noting that in early robotics papers, one could guess the research university simply by the background color of the lab. Simulation has yet to solve this deep generalization issue.
- Compliance with Human Preferences: Robots must perform tasks safely and in alignment with human expectations. For example, a robot should know to move a laptop out of the way before carrying a full cup of coffee over a desk. These subtle, task-dependent safety preferences are incredibly difficult to program into a simulator's reward function.
An Alternative Path: Real-World and Passive Data
Instead of engineering perfect simulators, the paper proposes leveraging the abundance of human activity data. The author points to recent successes in video pre-training, where robots learn useful manipulation priors directly from passive web videos (like YouTube tutorials). Because human preferences and safety constraints are implicitly present in videos of humans acting in the real world, this data provides a richer, more accurate training signal.
Additionally, the paper champions democratized real-world data collection. Tools like ALOHA enable high-quality real-world data gathering via low-cost teleoperation, and initiatives like the Open-X Embodiment project demonstrate the viability of training across diverse, real-world robot datasets.
Limitations and the Road Ahead
The author acknowledges that simulation is not entirely obsolete; it remains an excellent tool for prototyping algorithms, such as reinforcement learning agents for games. However, transitioning fully to a "Real-World" learning path presents its own hurdles:
- The Human-Robot Gap: Learning directly from human video is complicated by the "correspondence problem"—human hands and robot grippers have fundamentally different embodiments.
- Data Scarcity: While passive web video is abundant, high-quality, robot-specific action data remains harder to acquire than simulated data.
- Safety Risks: Training directly in the physical world carries inherent risks of hardware damage that simulation safely avoids.
Ultimately, the paper serves as a call to action. It urges the robotics community to redirect its collective resources away from building a "Matrix-like" perfect simulation of reality, and toward developing foundation models grounded in the vast, messy, and rich reality of human data.
Summary updated