TL;DR
RoboWorM is a new "deployment-aware" benchmark designed to evaluate robotic video world models (VWMs) based on physical reality rather than just visual aesthetics. By leveraging hardware priors like kinematic trees and camera configurations, it introduces two VLM-free metrics—KIVA and RIGIS—that outperform standard AI judges in detecting physical hallucinations and multi-view inconsistencies.
What Problem Does This Paper Address?
Video World Models (VWMs) are increasingly used as neural simulators to train robots, but we lack reliable ways to tell if their "imaginations" are physically accurate. Currently, researchers rely on visual metrics like FVD or Vision-Language Models (VLMs) to judge video quality. However, VLMs often suffer from "optimism bias" and lack the 3D spatial reasoning required to notice if a robot's arm is stretching unnaturally or if two camera views are showing different versions of the same room.
Furthermore, for real-world robotics, these models need to be fast. While various inference acceleration techniques exist, no one has systematically checked if these speed-ups "silently" break the laws of physics in the generated videos. RoboWorM bridges this gap by providing a framework to audit the trade-off between computational efficiency and physical fidelity.
Methods
The authors introduce two primary deterministic metrics that bypass the need for subjective VLM judges:
- KIVA (Kinematic Invariant Violation Assessment): This metric focuses on intra-view kinematic stability. It uses a diagnostic diffusion model to predict 3D robot keypoints from video frames. By comparing these to the robot's known "bone lengths" (rigidity) and checking for abrupt, non-physical movements (jerk), it identifies kinematic hallucinations.
- RIGIS (Robotic Inter-view Geometry & Intrinsic-Extrinsic Stability): This metric evaluates multi-view consistency. It uses known camera extrinsics and intrinsics to verify that synchronized views form a unified 3D scene. If a pixel in View A doesn't map correctly to View B, RIGIS flags it as a geometric failure.
The paper also conducts a massive audit of 9 acceleration methods (including token merging, feature caching, and sparse attention) across three leading world models: DreamGen, DreamDojo, and Ctrl-World.
Datasets
RoboWorM curates a diverse evaluation suite of 1,605 real-world episodes and over 1,200 prompts.
- Embodiments: Covers 5 distinct robot platforms across single-arm (ARX5, Franka Panda), bimanual (ALOHA), and humanoid (AIRBOT MK2, Fourier GR-1) morphologies.
- Task Dynamics: Includes both Markovian (short-horizon, e.g., pick and place) and non-Markovian (long-horizon, multi-step) tasks.
- Views: Supports 1–4 synchronized camera views.
Key Insights and Ideas
- Hardware Priors are Superior Judges: The "aha moment" here is that we don't need to guess if a video looks "right." Since we know the robot's physical dimensions and where the cameras are, we can use those invariants as a ground-truth "ruler" to measure model errors.
- The "Silent" Failure of Acceleration: The study reveals that many acceleration techniques maintain high visual quality (they look "clean" to the human eye) while significantly violating kinematic constraints. A model might run 2x faster but produce a robot that "teleports" its gripper—a failure standard metrics miss.
- VLM Inadequacy: Frontier VLMs (like Gemini or Qwen) struggle with strict 3D reasoning. KIVA and RIGIS align much more closely with expert human annotations because they are grounded in geometry rather than semantic "vibes."
Key Results
- Better Alignment: KIVA improves hallucination detection alignment with human labels by 25.8% relative to the strongest VLMs. RIGIS achieves a 130% higher Spearman correlation with human preferences for multi-view consistency.
- Efficiency vs. Quality: The audit identified WorldCache as a top-performing acceleration method, achieving a 2.3× maximum speed-up with minimal loss in physical integrity.
- Model Performance: Ctrl-World was found to dominate multi-view tasks due to its explicit spatial-temporal processing, while DreamDojo led in single-view tracks.
Limitations and Future Work
The authors acknowledge several areas for future improvement:
- Contact Physics: KIVA and RIGIS currently focus on rigid-body motion and do not verify contact-level validity (e.g., whether a gripper is actually applying the right force to an object).
- Deformable Objects: The metrics are designed for rigid robot links and do not yet capture the dynamics of deformable materials like cloth or liquids.
- Internal Auditing: The current audit measures output-level degradation but doesn't pinpoint which specific neural layers are responsible for losing physical fidelity during acceleration.
Future work will likely involve integrating full physics simulators into the evaluation loop to address these contact-level and material-specific limitations.
Summary updated