LynnReal-Hand: Unlocking World Interaction with Hand Control

Shaohao Rui, Chaoyang Zhang, Xiaofeng Mao, Weijie MaarXiv 2026 · arXiv

TL;DR

LynnReal-Hand is a high-fidelity, hand-controlled video world model that enables real-time interaction with virtual environments through articulated 3D hand motion. By combining a novel "Interaction-Preserving Training" strategy with a highly optimized "Ultra-Fast-VAE" decoder, the system achieves 18 FPS streaming generation on a single GPU with state-of-the-art hand-following accuracy.

What Problem Does This Paper Address?

While interactive video world models have made strides in responding to text prompts and camera movements, they struggle with direct object manipulation. Hands are the most natural interface for interaction, but modeling them in video generation is notoriously difficult for two reasons:

  1. Scale Mismatch: Hands and their contact points occupy only a tiny fraction of a video frame. Standard global training objectives (like mean squared error on pixels) tend to "wash out" these critical details, leading to blurry fingers or objects that don't react correctly to touch.
  2. Computational Latency: Real-time interaction requires extremely low latency. Generating high-resolution video that respects complex hand-object dynamics typically requires many denoising steps and heavy decoders, making "streaming" interaction nearly impossible on current hardware.

LynnReal-Hand tackles these by focusing on "interaction fidelity" throughout the entire training and inference pipeline.

Methods

The authors introduce Interaction-Preserving Training (IPT), a three-stage pipeline designed to maintain fine-grained control from pretraining to real-time deployment:

  1. Stage I: Hand-Aware Pretraining: The model (a 33B Diffusion Transformer) is trained with localized losses. Beyond standard flow matching, it uses latent ROI, perceptual (LPIPS), and semantic (DINOv2) constraints specifically on the hand regions to ensure the model learns precise finger articulation.
  2. Stage II: Causal Flow-Map Adaptation: To reduce sampling steps, the model is adapted into a few-step (3-step) causal generator. It uses a "two-chunk causal window" (W2) where the model generates new video segments while conditioning on its own previously generated history.
  3. Stage III: Region-Aware Dual-Score Distillation: To fix errors that accumulate during long rollouts, the authors use a "teacher" model to guide a "student." Crucially, they use two teachers: a video teacher for motion and an image teacher for background detail. They apply a spatial mask so the image teacher doesn't override the hand poses prescribed by the control signal.
Overview of the LynnReal-Hand training stages and architecture.

To solve the decoding bottleneck, they propose Ultra-Fast-VAE. This is a compressed Transformer-based decoder that uses structured pruning and "operator distillation" to match the quality of much larger models while running up to 17.8x faster.

Datasets

  • 100K Interaction Clips: A newly curated dataset of 100,000 video clips featuring diverse real and synthetic environments, all with temporally aligned hand trajectories, camera poses, and text descriptions.
  • HandBench: A new standardized benchmark consisting of 200 sequences (including out-of-distribution samples) specifically designed to evaluate hand-following accuracy and visual consistency in interactive models.

Key Insights and Ideas

  • High-Noise Guidance is Better for Dynamics: During distillation, evaluating the video teacher at high noise levels (e.g., $\sigma=0.90$) helps the model focus on correcting large-scale interaction errors rather than sweating minor texture discrepancies.
  • Spatial Separation in Distillation: A major "aha" moment was realizing that generic image priors (used to sharpen backgrounds) often conflict with specific hand-pose controls. By masking the image teacher's influence to only the background, they preserved both visual sharpness and control fidelity.
Visualization showing how high-noise video guidance and masked image guidance improve different aspects of the generation.
  • VAE Compression via Operator Distillation: Instead of just matching the final image output, they forced the compressed VAE's internal attention and feed-forward layers to match the behavior of the full-sized teacher, which was critical for preserving fine finger textures.

Key Results

  • State-of-the-Art Accuracy: On HandBench, LynnReal-Hand-Base achieved a MANO Dice score of 93.02, significantly outperforming existing baselines in how accurately the generated video follows the input hand mesh.
  • Real-Time Performance: The "Fast" version of the model, running at 540p on a single NVIDIA H200, reaches 18.03 FPS with a latency of just 0.741 seconds.
  • Decoder Efficiency: The Ultra-Fast-VAE achieves a PSNR of 34.43 dB while reducing latency from 831ms (Native H3) to just 46.77ms.
Qualitative comparison of LynnReal-Hand against other baselines, showing superior finger-object contact and scene stability.

Limitations and Future Work

  • Short-Term Memory: The current two-chunk causal window (W2) prioritizes speed but lacks a long-term 3D memory. This can lead to "object permanence" issues where an object might change or disappear if it leaves and then re-enters the camera's view.
  • Physical Priors: While the model follows kinematic hand controls well, it doesn't explicitly model physics (like gravity or friction). The authors suggest that coupling these kinematic controls with physical contact priors is a key next step, especially for applications in dexterous robot learning.

Summary updated

Abstract

Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.
BibTeX
@misc{rui2026lynnreal,
  author = {Shaohao Rui and Chaoyang Zhang and Xiaofeng Mao and Weijie Ma},
  title = {{LynnReal-Hand: Unlocking World Interaction with Hand Control}},
  year = {2026},
  eprint = {2611.21479},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21479},
}
Figure