Abstract
Evaluating robot control policies remains a significant bottleneck in robotics research due to the high costs of real-world testing and the complexity of hand-crafting traditional simulators. We present WorldGym, a novel evaluation framework that utilizes action-conditioned video generation models as "generative simulators." By "hallucinating" robot-environment interactions and employing Vision-Language Models (VLMs) as automated reward functions, WorldGym can predict real-world policy performance with high accuracy ($r = 0.78$) without physical deployment. This approach enables rapid, scalable, and safe evaluation of Vision-Language-Action (VLA) models.
Introduction
Traditional robot evaluation relies on two primary methods: expensive real-world trials or manually designed simulators like MuJoCo. While simulators are safer, they often suffer from a "sim-to-real gap," particularly when modeling complex physics like soft objects or messy, real-world environments. WorldGym addresses this by learning the laws of physics directly from diverse video data, creating a sandbox that can flexibly test policies across various morphologies and tasks.
The WorldGym Framework
WorldGym integrates three core components to create a closed-loop evaluation environment:
- The World Model: A Latent Diffusion Transformer (DiT) trained using "Diffusion Forcing." This allows the model to generate variable-length video sequences that match the specific "action chunks" predicted by a robot policy.
- Action Conditioning: The model is conditioned on 7D action vectors (6D end-effector position and 1D gripper state), allowing it to visually react to the policy's commands.
- VLM as Reward: The generated "hallucinated" rollout is passed to a VLM (e.g., GPT-4o), which acts as an automated referee to determine task success based on visual evidence.
Generative Simulation Fidelity
A critical strength of WorldGym is its ability to represent complex interactions that are difficult to code in traditional physics engines. By training on the Open-X Embodiment dataset (including Bridge V2 and RT-1), the world model learns a general representation of physics. This enables the simulation of challenging tasks, such as manipulating deformable objects, with high visual fidelity compared to ground-truth data.
Key Results and Insights
- Predictive Accuracy: Success rates in WorldGym show a strong Pearson correlation of 0.78 with real-world results, with predicted success rates typically within 3.3% of actual values.
- Ranking Preservation: The framework correctly identifies the relative performance of different models (e.g., OpenVLA > Octo > RT-1-X), making it a reliable tool for checkpoint selection during training.
- Out-of-Distribution (OOD) Testing: WorldGym allows researchers to test robustness by using image-editing models to modify initial frames—adding distractors or changing object colors—enabling OOD testing in seconds rather than days.
- Efficiency: Evaluations that would take days in a physical lab can be completed in under an hour on a single GPU, providing a 2.8x speedup over fixed-window generation models.
Conclusion
WorldGym serves as a practical and reproducible "sanity check" for robot evaluation. While challenges remain in capturing extremely fine-grained object interactions and long-term temporal consistency, the framework provides a scalable path toward testing general-purpose robot agents in complex, diverse environments before moving to physical deployment.
Summary updated