TL;DR
TeleWorld introduces a real-time, 18-billion-parameter 4D world model that unifies video generation and dynamic scene reconstruction into a closed-loop system. By combining hierarchical planning with efficient distribution distillation and distributed training optimizations, it achieves state-of-the-art consistency and interactive 8 FPS generation on highly practical hardware budgets.
What Problem Does This Paper Address?
While modern video generation models produce visually stunning clips, they fall short of being true "world models." They lack real-time interactivity, suffer from error accumulation over long horizons (leading to visual drift, color shifts, or "forgetting" scene geometry), and fail to maintain a persistent memory of moving objects across space and time (4D memory).
Furthermore, scaling these models to over 10 billion parameters makes real-time inference and distillation training computationally prohibitive for most researchers, often requiring massive, inaccessible GPU clusters. TeleWorld tackles these challenges by bridging the gap between video generation, 3D reconstruction, and real-time execution.
Methods
The authors introduce several novel architectural and system-level components to build a practical 4D world model:
-
"Generation-Reconstruction-Guidance" Loop: Instead of relying solely on implicit latent states, TeleWorld continuously reconstructs generated video frames into an explicit 4D spatio-temporal point cloud field. This persistent 4D memory is rendered on the fly and used as a structural guide for subsequent frame generation, alongside user keyboard inputs (WASD/arrows) mapped to camera poses.
Structure of the TeleWorld closed-loop framework -
Macro-from-Micro Planning (MMPL): To prevent long-term temporal drift, TeleWorld uses a hierarchical approach. Micro-planning jointly optimizes a sparse set of key anchor frames within short segments to eliminate frame-to-frame cumulative drift. Macro-planning then chains these segments autoregressively across the timeline.
The hierarchical Macro-from-Micro Planning (MMPL) framework -
Large-Scale Distribution Matching Distillation (DMD): To achieve real-time speeds, the model leverages DMD. To overcome the massive memory footprint of running three 10B+ parameter models simultaneously (generator, teacher, and critic), the authors sharded the generator's KV cache using context parallelism and distributed the components across disjoint GPU groups using Ray.
- Streaming Infrastructure: The pipeline integrates a chunk-wise Streamed VAE (processing 4-frame chunks to minimize "time to first frame") and a locality-constrained sparse attention mechanism for real-time video super-resolution.
Datasets
- TeleWorld-500K: A newly introduced, curated dataset containing 500,000 high-quality video clips with controllable camera movements and dynamic interactions. It features comprehensive 4D annotations, including motion object masks, 3D trajectories, camera poses, and detailed semantic descriptions generated via Qwen-2.5-VL.
- WorldScore Benchmark: The primary evaluation protocol used to measure "world generation" capabilities across both static and dynamic settings.
Key Insights and Ideas
- Explicit 4D Memory Beats Implicit Latents: Reconstructing an explicit 4D point cloud on the fly anchors the model's environment state. This prevents the "world collapse" or semantic drift typical of pure autoregressive video models over long durations.
- Decoupling Training Components via Pipeline Parallelism: The memory footprint of large-scale DMD distillation normally requires over 64 H100 GPUs. Separating the generator, teacher, and critic onto disjoint GPU groups—and carefully overlapping their execution schedules—minimizes pipeline bubbles and cuts hardware requirements in half.
- Hierarchical Planning Bounds Error Propagation: Jointly optimizing key anchor frames within a segment bounds residual errors locally, transforming fragile frame-by-frame error accumulation into a highly stable segment-by-segment progression.
Key Results
- Benchmark Leaderboard: TeleWorld ranks 1st on the WorldScore benchmark, achieving a WorldScore-Static of 78.23 (outperforming the best baseline, Voyager, by 0.61) and a WorldScore-Dynamic of 66.73 (shattering the previous best baseline, CogVideoX-I2V, by 7.61). It is the only model to simultaneously rank first in both tracks.
- Real-Time Efficiency: The system delivers a sustained throughput exceeding 32 FPS for the TeleWorld-1.3B variant and 8 FPS for the full TeleWorld-18B model on NVIDIA H100 GPUs.
- Training Feasibility: Thanks to system-level optimizations, the 18B model was successfully trained for DMD long-video distillation using only 32 H100 GPUs instead of the typical 64+.
- High-Resolution Output: The integrated super-resolution module successfully upscales video latents to a crisp 960×1760 resolution at approximately 17 FPS.
Limitations and Future Work
- Interaction Latency: To minimize peak memory and latency, the model employs a "Minimum Memory Peak Prediction" strategy that introduces a feedback latency of approximately one second (three latent chunks). Consequently, immediate user keyboard inputs are not reflected instantaneously.
- Dynamic Sparsity: Because the 4D reconstruction module relies strictly on sparse pre-planning anchor frames, the rendered dynamic content can be highly sparse. The framework relies heavily on macro-planning to predict and smooth out subsequent dynamic regions.
Summary updated