TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model

Yabo Chen, Yuanzhi Liang, Jiepeng Wang, Tingxi Chen, Junfei Cheng, Zixiao Gu, Yuyang Huang, Zicheng Jiang, Wei Li, Tian Li, Zuoxin Li, Guangce Liu, Jialun Liu, Junqi Liu, Haoyuan Wang, Qizhen Weng, Xuan'er Wu, Xunzhi Xiang, Xiaoyan Yang, Xin Zhang, Shiwen Zhang, Junyu Zhou, Chengcheng Zhou, Haibin Huang, Chi Zhang, Xuelong LiarXiv 2025 · arXiv

TL;DR

TeleWorld introduces a real-time, 18-billion-parameter 4D world model that unifies video generation and dynamic scene reconstruction into a closed-loop system. By combining hierarchical planning with efficient distribution distillation and distributed training optimizations, it achieves state-of-the-art consistency and interactive 8 FPS generation on highly practical hardware budgets.

What Problem Does This Paper Address?

While modern video generation models produce visually stunning clips, they fall short of being true "world models." They lack real-time interactivity, suffer from error accumulation over long horizons (leading to visual drift, color shifts, or "forgetting" scene geometry), and fail to maintain a persistent memory of moving objects across space and time (4D memory).

Furthermore, scaling these models to over 10 billion parameters makes real-time inference and distillation training computationally prohibitive for most researchers, often requiring massive, inaccessible GPU clusters. TeleWorld tackles these challenges by bridging the gap between video generation, 3D reconstruction, and real-time execution.

Methods

The authors introduce several novel architectural and system-level components to build a practical 4D world model:

  • "Generation-Reconstruction-Guidance" Loop: Instead of relying solely on implicit latent states, TeleWorld continuously reconstructs generated video frames into an explicit 4D spatio-temporal point cloud field. This persistent 4D memory is rendered on the fly and used as a structural guide for subsequent frame generation, alongside user keyboard inputs (WASD/arrows) mapped to camera poses.

    Structure of the TeleWorld closed-loop framework
  • Macro-from-Micro Planning (MMPL): To prevent long-term temporal drift, TeleWorld uses a hierarchical approach. Micro-planning jointly optimizes a sparse set of key anchor frames within short segments to eliminate frame-to-frame cumulative drift. Macro-planning then chains these segments autoregressively across the timeline.

    The hierarchical Macro-from-Micro Planning (MMPL) framework
  • Large-Scale Distribution Matching Distillation (DMD): To achieve real-time speeds, the model leverages DMD. To overcome the massive memory footprint of running three 10B+ parameter models simultaneously (generator, teacher, and critic), the authors sharded the generator's KV cache using context parallelism and distributed the components across disjoint GPU groups using Ray.

  • Streaming Infrastructure: The pipeline integrates a chunk-wise Streamed VAE (processing 4-frame chunks to minimize "time to first frame") and a locality-constrained sparse attention mechanism for real-time video super-resolution.

Datasets

  • TeleWorld-500K: A newly introduced, curated dataset containing 500,000 high-quality video clips with controllable camera movements and dynamic interactions. It features comprehensive 4D annotations, including motion object masks, 3D trajectories, camera poses, and detailed semantic descriptions generated via Qwen-2.5-VL.
  • WorldScore Benchmark: The primary evaluation protocol used to measure "world generation" capabilities across both static and dynamic settings.

Key Insights and Ideas

  • Explicit 4D Memory Beats Implicit Latents: Reconstructing an explicit 4D point cloud on the fly anchors the model's environment state. This prevents the "world collapse" or semantic drift typical of pure autoregressive video models over long durations.
  • Decoupling Training Components via Pipeline Parallelism: The memory footprint of large-scale DMD distillation normally requires over 64 H100 GPUs. Separating the generator, teacher, and critic onto disjoint GPU groups—and carefully overlapping their execution schedules—minimizes pipeline bubbles and cuts hardware requirements in half.
  • Hierarchical Planning Bounds Error Propagation: Jointly optimizing key anchor frames within a segment bounds residual errors locally, transforming fragile frame-by-frame error accumulation into a highly stable segment-by-segment progression.

Key Results

  • Benchmark Leaderboard: TeleWorld ranks 1st on the WorldScore benchmark, achieving a WorldScore-Static of 78.23 (outperforming the best baseline, Voyager, by 0.61) and a WorldScore-Dynamic of 66.73 (shattering the previous best baseline, CogVideoX-I2V, by 7.61). It is the only model to simultaneously rank first in both tracks.
  • Real-Time Efficiency: The system delivers a sustained throughput exceeding 32 FPS for the TeleWorld-1.3B variant and 8 FPS for the full TeleWorld-18B model on NVIDIA H100 GPUs.
  • Training Feasibility: Thanks to system-level optimizations, the 18B model was successfully trained for DMD long-video distillation using only 32 H100 GPUs instead of the typical 64+.
  • High-Resolution Output: The integrated super-resolution module successfully upscales video latents to a crisp 960×1760 resolution at approximately 17 FPS.

Limitations and Future Work

  • Interaction Latency: To minimize peak memory and latency, the model employs a "Minimum Memory Peak Prediction" strategy that introduces a feedback latency of approximately one second (three latent chunks). Consequently, immediate user keyboard inputs are not reflected instantaneously.
  • Dynamic Sparsity: Because the 4D reconstruction module relies strictly on sparse pre-planning anchor frames, the rendered dynamic content can be highly sparse. The framework relies heavily on macro-planning to predict and smooth out subsequent dynamic regions.

Summary updated

Abstract

World models aim to endow AI systems with the ability to represent, generate, and interact with dynamic environments in a coherent and temporally consistent manner. While recent video generation models have demonstrated impressive visual quality, they remain limited in real-time interaction, long-horizon consistency, and persistent memory of dynamic scenes, hindering their evolution into practical world models. In this report, we present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system. TeleWorld introduces a novel generation-reconstruction-guidance paradigm, where generated video streams are continuously reconstructed into a dynamic 4D spatio-temporal representation, which in turn guides subsequent generation to maintain spatial, temporal, and physical consistency. To support long-horizon generation with low latency, we employ an autoregressive diffusion-based video model enhanced with Macro-from-Micro Planning (MMPL)--a hierarchical planning method that reduces error accumulation from frame-level to segment-level-alongside efficient Distribution Matching Distillation (DMD), enabling real-time synthesis under practical computational budgets. Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible systems. Extensive experiments demonstrate that TeleWorld achieves strong performance in both static and dynamic world understanding, long-term consistency, and real-time generation efficiency, positioning it as a practical step toward interactive, memory-enabled world models for multimodal generation and embodied intelligence.
BibTeX
@misc{chen2025teleworld,
  author = {Yabo Chen and Yuanzhi Liang and Jiepeng Wang and Tingxi Chen and Junfei Cheng and Zixiao Gu and Yuyang Huang and Zicheng Jiang and Wei Li and Tian Li and Weichen Li and Zuoxin Li and Guangce Liu and Jialun Liu and Junqi Liu and Haoyuan Wang and Qizhen Weng and Xuan'er Wu and Xunzhi Xiang and Xiaoyan Yang and Xin Zhang and Shiwen Zhang and Junyu Zhou and Chengcheng Zhou and Haibin Huang and Chi Zhang and Xuelong Li},
  title = {{TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model}},
  year = {2025},
  eprint = {2601.00051},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.00051},
}
Figure