TL;DR
WorldArena 2.0 expands embodied world model benchmarking across three critical dimensions: modality (integrating visuotactile sensing), functionality (evaluating world models as active, online reinforcement learning environments), and platform (bridging simulation to real-world hardware). By evaluating 12 state-of-the-art models, the benchmark reveals that while visual generation quality is rapidly improving, a massive sim-to-real usability gap still hinders actual physical deployment.
What Problem Does This Paper Address?
An embodied world model acts as an AI agent's internal simulator, allowing it to predict the consequences of its actions and plan ahead. However, existing benchmarks for these models suffer from three major blind spots:
- Vision-Only Focus: They ignore tactile feedback, which is essential for contact-rich tasks involving friction, force, and slip.
- Offline Evaluation: They mostly test models on static, open-loop planning tasks rather than evaluating whether a world model can serve as an interactive environment to train a reinforcement learning (RL) policy from scratch.
- Simulator Lock-in: Evaluation happens almost exclusively in pristine virtual environments, leaving it unclear if high performance translates to messy, real-world robotics.
WorldArena 2.0 addresses these gaps by providing a standardized, multi-platform testing ground that pushes world models beyond simple video generation and into true physical interaction.
Methods
The authors systematically upgrade the evaluation framework across three axes:
- Visuotactile Integration: Instead of altering underlying model architectures, they introduce a modular pipeline. A specialized tactile VAE encodes tactile deformation maps into the video latent space of an existing world model. A two-stream model then performs synchronized denoising for both video and tactile predictions, which an action diffusion head uses to infer robot actions.
- World Models as RL Environments: They build a closed-loop pipeline where the world model completely replaces the physics simulator during RL training. The policy agent interacts recursively with the world model, which generates the next observation and a reward (via a learned reward model) to optimize the policy using algorithms like Group Relative Policy Optimization (GRPO).
- Cross-Embodiment Pipeline: They establish a graded evaluation pipeline that moves from heavily randomized simulation to structured knowledge transfer, and finally to real-world physical execution.
Datasets and Platforms
WorldArena 2.0 benchmarks models across four distinct setups:
- UniVTAC Simulator: Used for evaluating visuotactile capabilities on contact-intensive tasks like Insert HDMI and Lift Bottle.
- RoboTwin 2.0: A bimanual simulation environment with 731 objects used to test out-of-distribution generalization via tasks like Adjust Bottle and Click Bell.
- LIBERO: A simulation suite based on Robosuite used to evaluate knowledge transfer on tasks like Turn on the Stove.
- AgileX Split-Type ALOHA: A real-world bimanual robot setup used as the ultimate physical reality check for tasks like Wipe Table and Pour Water.
Key Insights and Ideas
- Visual Realism $\neq$ Physical Validity: High-end commercial video generation models (like Veo 3.1 and Wan 2.6) dominate visual quality metrics. However, their advantage shrinks drastically when evaluating physics adherence, where robotics-specific models like CtrlWorld and IRASim achieve superior trajectory accuracy.
- Generalist Models Possess Strong Cross-Modal Priors: Surprisingly, the general-purpose video model Wan 2.2 achieved the highest tactile prediction quality, outperforming specialized embodied models. This suggests that large-scale video pre-training instills rich, latent physical priors that easily align with non-visual modalities like touch.
- The Sim-to-Real Mirage: While task success rates correlate positively between different simulators, that correlation plummets when compared against real-world performance. Simulation performance is simply not a reliable proxy for real-world deployment.
Key Results
- Visuotactile Success: Benefiting from its superior tactile prediction, Wan 2.2 achieved a 100% success rate on the contact-rich Insert HDMI task in simulation, whereas the vision-only baseline struggled.
- World Models as Simulators: When training RL agents entirely inside world models, WoVR and CtrlWorld achieved performance comparable to training on the ground-truth physics simulator for short-horizon tasks (e.g., Click Bell).
- The Real-World Crash: In real-world testing on the ALOHA platform, performance dropped sharply. For the Pour Water task, the best-performing model (Vidar) only achieved a 40% success rate, while several prominent models dropped to 0%, highlighting the severity of compounding errors in the real world.
Limitations and Future Work
A major bottleneck identified by the benchmark is long-horizon planning. In the Lift Bottle task—which requires sustained, high-frequency force feedback over a long period—every single tested world model failed completely (0% success rate) due to compounding prediction errors over time.
Additionally, the massive sim-to-real gap remains unsolved. The authors note that future iterations of WorldArena will focus on expanding to even more sensory modalities (like audio), increasing task complexity, and developing techniques to mitigate the compounding errors that cause models to fail when deployed on physical robots.
Summary updated