RoboWorM: Evaluating Movement Realism, Consistency and Acceleration Affects in Embodied Video World Models

Doanh Le Thien, Viet-Thanh Nguyen, Pham Tri Quang, Nhi Pham, Christopher Wewer, Minh-Loi Nguyen, Hoang H. Tran, Tho Le, Tung M. Luu, Dung D. Le, Vu N. Duong, Jonas Fischer, Ngan Le, An Thai Le, Daniel Sonntag, Gim Hee Lee, James Zou, Mathias Niepert, Jan Peters, Duy M. H. Nguyen, Ngo Anh VienarXiv 2026 · arXiv

TL;DR

RoboWorM is a new "deployment-aware" benchmark designed to evaluate robotic video world models (VWMs) based on physical reality rather than just visual aesthetics. By leveraging hardware priors like kinematic trees and camera configurations, it introduces two VLM-free metrics—KIVA and RIGIS—that outperform standard AI judges in detecting physical hallucinations and multi-view inconsistencies.

What Problem Does This Paper Address?

Video World Models (VWMs) are increasingly used as neural simulators to train robots, but we lack reliable ways to tell if their "imaginations" are physically accurate. Currently, researchers rely on visual metrics like FVD or Vision-Language Models (VLMs) to judge video quality. However, VLMs often suffer from "optimism bias" and lack the 3D spatial reasoning required to notice if a robot's arm is stretching unnaturally or if two camera views are showing different versions of the same room.

Furthermore, for real-world robotics, these models need to be fast. While various inference acceleration techniques exist, no one has systematically checked if these speed-ups "silently" break the laws of physics in the generated videos. RoboWorM bridges this gap by providing a framework to audit the trade-off between computational efficiency and physical fidelity.

Methods

The authors introduce two primary deterministic metrics that bypass the need for subjective VLM judges:

  • KIVA (Kinematic Invariant Violation Assessment): This metric focuses on intra-view kinematic stability. It uses a diagnostic diffusion model to predict 3D robot keypoints from video frames. By comparing these to the robot's known "bone lengths" (rigidity) and checking for abrupt, non-physical movements (jerk), it identifies kinematic hallucinations.
  • RIGIS (Robotic Inter-view Geometry & Intrinsic-Extrinsic Stability): This metric evaluates multi-view consistency. It uses known camera extrinsics and intrinsics to verify that synchronized views form a unified 3D scene. If a pixel in View A doesn't map correctly to View B, RIGIS flags it as a geometric failure.

The paper also conducts a massive audit of 9 acceleration methods (including token merging, feature caching, and sparse attention) across three leading world models: DreamGen, DreamDojo, and Ctrl-World.

Datasets

RoboWorM curates a diverse evaluation suite of 1,605 real-world episodes and over 1,200 prompts.

  • Embodiments: Covers 5 distinct robot platforms across single-arm (ARX5, Franka Panda), bimanual (ALOHA), and humanoid (AIRBOT MK2, Fourier GR-1) morphologies.
  • Task Dynamics: Includes both Markovian (short-horizon, e.g., pick and place) and non-Markovian (long-horizon, multi-step) tasks.
  • Views: Supports 1–4 synchronized camera views.
Figure 1: Overview of RoboWorM, a deployment-aware benchmark auditing the physical fidelity and computational efficiency of robotic video world models. (a) Dataset Collection: 1,600+ real- world episodes and 1,200+ prompts across 5 embodiments (single-arm, bimanual, humanoid), 1–4 views, and diverse task horizons. (b) Physics Audit: We propose two deterministic, VLM-free metrics leveraging hardware priors: KIVA for kinematic stability and RIGIS for 3D geometric consistency. (c) Inference Acceleration: we evaluate 3 world models across 9 acceleration meth- ods (29k rollouts), examining how speed-ups compromise deployment-critical physical constraints. Project page: https://roboworm.github.io/roboworm/.

Key Insights and Ideas

  • Hardware Priors are Superior Judges: The "aha moment" here is that we don't need to guess if a video looks "right." Since we know the robot's physical dimensions and where the cameras are, we can use those invariants as a ground-truth "ruler" to measure model errors.
  • The "Silent" Failure of Acceleration: The study reveals that many acceleration techniques maintain high visual quality (they look "clean" to the human eye) while significantly violating kinematic constraints. A model might run 2x faster but produce a robot that "teleports" its gripper—a failure standard metrics miss.
  • VLM Inadequacy: Frontier VLMs (like Gemini or Qwen) struggle with strict 3D reasoning. KIVA and RIGIS align much more closely with expert human annotations because they are grounded in geometry rather than semantic "vibes."

Key Results

  • Better Alignment: KIVA improves hallucination detection alignment with human labels by 25.8% relative to the strongest VLMs. RIGIS achieves a 130% higher Spearman correlation with human preferences for multi-view consistency.
  • Efficiency vs. Quality: The audit identified WorldCache as a top-performing acceleration method, achieving a 2.3× maximum speed-up with minimal loss in physical integrity.
  • Model Performance: Ctrl-World was found to dominate multi-view tasks due to its explicit spatial-temporal processing, while DreamDojo led in single-view tracks.
Overview of RoboWorM's three-pillar approach: Dataset, Physics Audit, and Acceleration

Limitations and Future Work

The authors acknowledge several areas for future improvement:

  • Contact Physics: KIVA and RIGIS currently focus on rigid-body motion and do not verify contact-level validity (e.g., whether a gripper is actually applying the right force to an object).
  • Deformable Objects: The metrics are designed for rigid robot links and do not yet capture the dynamics of deformable materials like cloth or liquids.
  • Internal Auditing: The current audit measures output-level degradation but doesn't pinpoint which specific neural layers are responsible for losing physical fidelity during acceleration.

Future work will likely involve integrating full physics simulators into the evaluation loop to address these contact-level and material-specific limitations.

Summary updated

Abstract

Video world models (VWMs) are rapidly emerging as scalable neural simulators for robotic manipulation. However, most benchmarks only evaluate them as open-world video generators, using Vision-Language Models (VLMs) that lack strict 3D spatial and kinematic reasoning. To bridge this gap, we introduce RoboWorM, a comprehensive, deployment-aware benchmark that leverages known hardware priors such as robot kinematic trees and fixed camera configurations, to systematically examine structural and spatiotemporal fidelity. RoboWorM curates 1,605 episodes across single-arm, bimanual, and humanoid morphologies, spanning both Markovian and long-horizon non-Markovian tasks under single- and multi-view setups. To explicitly measure physical consistency, we propose two deterministic, VLM-free metrics: KIVA for intra-view kinematic stability, and RIGIS for multi-view geometric consistency. By exploiting intrinsic robotic invariants, KIVA improves hallucination detection alignment with human annotations by 25.8% relative to the strongest frontier VLMs, while RIGIS achieves a 130% higher Spearman correlation with human preferences. Furthermore, since real-world deployment requires low inference latency, we utilize RoboWorM to conduct the first large-scale audit of the quality-efficiency trade-off. Evaluating 9 state-of-the-art acceleration techniques across 3 leading world models (over 29,000 generated rollouts), we demonstrate how these acceleration methods can retain superficial visual quality while silently compromising deployment-critical physical constraints. Using our metrics, we achieve 2.3$\times$ maximum inference speed-up with minimal quality loss.
BibTeX
@misc{thien2026roboworm,
  author = {Doanh Le Thien and Viet-Thanh Nguyen and Pham Tri Quang and Nhi Pham and Christopher Wewer and Minh-Loi Nguyen and Hoang H. Tran and Tho Le and Tung M. Luu and Dung D. Le and Vu N. Duong and Jonas Fischer and Ngan Le and An Thai Le and Daniel Sonntag and Gim Hee Lee and James Zou and Mathias Niepert and Jan Peters and Duy M. H. Nguyen and Ngo Anh Vien},
  title = {{RoboWorM: Evaluating Movement Realism, Consistency and Acceleration Affects in Embodied Video World Models}},
  year = {2026},
  eprint = {2610.33807},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33807},
}
Figure