HandWorldBench: A Unified Benchmark for Hand-Centric World Models

Zhaolu Wang, Jiacheng Xu, Lihong Wang, Shuqing Wang, Jianjie Fang, Ziyou Wang, Xvyuan Liu, Rongze Tang, Yiding Ma, Xin Zhang, Jiawei Chen, Wei Wu, Chen Gao, Yong Li, Zhibo ChenarXiv 2026 · arXiv

TL;DR

HandWorldBench is a new unified benchmark designed to rigorously evaluate how "world models" handle the intricate physics of hand-object interactions across human, simulated, and real robotic hands. The study reveals a critical "decoupling" in current AI: models that are excellent at following precise hand-pose instructions often fail miserably at predicting the actual physical consequences—like an object moving or changing shape—resulting from those actions.

What Problem Does This Paper Address?

While generative AI has made massive strides in creating realistic videos, "hand-centric" world models—AI that can predict how a hand's movement will affect the world—remain a "boss level" challenge. Hands are complex, high-degree-of-freedom manipulators that cause frequent occlusions and fine-grained physical changes (like pinching a dumpling or turning a tiny screw).

Existing benchmarks are often fragmented, focusing either on purely visual quality or on specific robotic control tasks. There has been no unified way to measure if a model truly understands the "physics of the hand" across different embodiments (human vs. robot) and whether it can maintain temporal and physical consistency during complex tasks.

Methods

The authors developed a comprehensive evaluation framework consisting of:

  1. Unified Control Interface: To compare different models fairly, they mapped various control signals (3D keypoints for humans, joint states for robots) into a common format. This allows researchers to test models using their "native" inputs while evaluating them on the same tasks.
  2. Six-Dimensional Evaluation: Instead of a single accuracy score, they use 17 metrics across six key dimensions:
  3. Visual Quality: Perceptual realism.
  4. Hand Fidelity: Accuracy of hand poses and morphology.
  5. Appearance Consistency: Do objects and backgrounds stay the same over time?
  6. Instruction Following: Does the model actually perform the requested action?
  7. Physical Consistency: Do the resulting object trajectories and state changes make physical sense?
  8. Temporal Coherence: Does the model "remember" object states even after occlusions?
  9. HWBSCORE: A normalized aggregate metric (0–100) to rank model performance across all dimensions.
Figure 1. Overview of HandWorldBench. Left: 500 evaluation cases span human hands, simulated dexterous hands, and real dexterous hands. Center: seven task families encompass 22 task categories. Right: six evaluation dimensions capture complementary aspects of hand-centric generation and interaction prediction.

Datasets

The researchers built a massive data pipeline to support this benchmark:

  • The Corpus: Over 600 hours of raw egocentric (first-person) footage was processed.
  • Curation: A rigorous filtering and validation pipeline (checking for camera stability, hand visibility, and signal alignment) yielded 20,500 high-quality interaction clips.
  • The Benchmark Set: From this pool, 500 clips were manually selected for the final evaluation set. These cover 7 task families (e.g., Material Shaping, Tools & Mechanisms) and 22 task categories.
  • Embodiments: The set includes 322 human hand cases, 108 simulated dexterous hand cases, and 70 real-world robotic dexterous hand cases.
Figure 1. Overview of HandWorldBench. Left: 500 evaluation cases span human hands, simulated dexterous hands, and real dexterous hands. Center: seven task families encompass 22 task categories. Right: six evaluation dimensions capture complementary aspects of hand-centric generation and interaction prediction.

Key Insights and Ideas

  • The "Puppeteer" Problem: One of the most surprising findings is the negative correlation between "Pose Control" and "State Progression." Models that are the best at precisely drawing a hand in a specific pose often struggle the most to show that hand actually doing something to an object. They are good puppets but bad physicists.
  • Generalists vs. Specialists: General-purpose video generators (like HunyuanVideo 1.5) often outperformed specialized hand-control models on dexterous robotic tasks. This suggests that the broad physical priors learned from massive video datasets are currently more robust than the narrow, pose-conditioned training used for specialized models.
  • Memory is the Weak Link: Models frequently fail at "Temporal Coherence." For example, if a hand wipes a surface and briefly covers a pattern, the pattern might disappear or change entirely when the hand moves away.

Key Results

The authors evaluated 11 representative models.

  • Top Performer: JointControlVideo achieved the highest overall HWBSCORE of 55.51, showing a strong balance between following instructions and maintaining physical consistency.
  • Visual Leaders: Generalist models like HunyuanVideo 1.5 (52.22) and Wan2.2 (51.65) led in visual quality and appearance consistency but lagged in precise physical regulation.
  • Human Alignment: The automated HWBSCORE showed a very high correlation ($\rho = 0.964$) with human preference judgments, validating the benchmark's metrics.
Figure 3. Overall ranking and cross-embodiment profiles. Left: visual quality (VQ), hand fidelity (HF), appearance consistency (AC), instruction following (IF), physical consistency (PC), and temporal coherence (TC). Radar colors and markers identify models. For each category, the four radars share the maximum over eleven models and four evaluation sets as their outer radius. Right: raw HWBSCORE, colored by rank.

Limitations and Future Work

  • Task Duration: The current benchmark focuses on relatively short interaction clips (approx. 80 frames). Evaluating long-horizon tasks where errors can compound over minutes remains a future challenge.
  • Embodiment Diversity: While the benchmark covers human and dexterous robotic hands, expanding to more diverse grippers or multi-agent (two-handed) interactions is a logical next step.
  • Causal Inference: The authors note that while they can measure if a state change happened, deeper metrics are needed to evaluate if the model understands the causal link between a specific finger pressure and a specific object deformation.

Summary updated

Abstract

Recent advances in world models are opening new possibilities for fine-grained embodied tasks, with hand-object interaction offering a particularly demanding setting. Modeling these interactions involves capturing how hand actions change objects and how the resulting states persist over time. Yet systematic evaluation tailored to these hand-centric capabilities remains underdeveloped. To address this gap, we introduce HandWorldBench, a unified benchmark that systematically evaluates hand-centric world models across human hands, simulated dexterous hands, and real dexterous hands along six dimensions: visual quality, hand fidelity, appearance consistency, instruction following, physical consistency, and temporal coherence. To construct the benchmark, we develop a data curation pipeline that extracts hand-object interaction clips and screens them for quality, yielding 20,500 high-quality egocentric clips from over 600 hours of footage. We then manually select 500 clips covering seven task families for evaluation. Benchmarking eleven models under a unified protocol reveals that stronger hand-pose control does not consistently translate into coherent object-state evolution and persistence or robust cross-embodiment performance, with several specialized hand-control models underperforming general-purpose text/image-to-video generators on dexterous-hand tasks.
BibTeX
@misc{wang2026handworldbench,
  author = {Zhaolu Wang and Jiacheng Xu and Lihong Wang and Shuqing Wang and Jianjie Fang and Ziyou Wang and Xvyuan Liu and Rongze Tang and Yiding Ma and Xin Zhang and Jiawei Chen and Wei Wu and Chen Gao and Yong Li and Zhibo Chen},
  title = {{HandWorldBench: A Unified Benchmark for Hand-Centric World Models}},
  year = {2026},
  eprint = {2611.24731},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24731},
}
Figure