ScaleWorld: High-Resolution World Generation through Shared Prediction and Refinement

Yansong Qu, Wesker Zheng, Quanjian Song, Zhiyuan Zhang, Xiao Cao, Shengchuan Zhang, Liujuan CaoarXiv 2026 · arXiv

TL;DR

ScaleWorld is a unified causal video model that performs both low-resolution world prediction and high-resolution refinement within a single shared backbone. By utilizing an asymmetric causal memory and a novel distillation technique called Task-Asymmetric On-Policy Distillation (TA-OPD), it achieves state-of-the-art high-resolution video generation while reducing peak GPU memory by 21% and increasing throughput by 27% compared to traditional multi-model cascades.

What Problem Does This Paper Address?

Generating high-resolution (HR) video for "world models"—AI systems that simulate interactive environments—is notoriously expensive. Standard approaches typically use a coarse-to-fine pipeline: one model predicts the next frames at a low resolution (LR), and a second, dedicated "refiner" model upscales them.

While effective, this "cascade" approach is inefficient. It requires storing two sets of massive model weights, and both models often learn redundant visual priors about structure and motion. Furthermore, directly training a single model to handle both tasks often leads to task interference, where the gradients for prediction and refinement conflict, degrading overall performance.

Methods

ScaleWorld unifies these tasks into one shared DiT (Diffusion Transformer) backbone. The key innovations are:

  1. Shared Backbone with Separate States: The model alternates between two roles: World Prediction (WM) and Refinement (Ref). While weights are shared, the model maintains separate "causal states" (KV caches) for each task.
  2. Asymmetric Causal Memory:
    • WM uses a long-horizon "clean" LR history to maintain global scene consistency.
  3. Ref uses a short-horizon "noisy-forward" HR history. Instead of performing an extra forward pass to "clean" the history (re-encoding), it reuses the KV states generated during the denoising process, drastically saving computation.
  4. Task-Asymmetric On-Policy Distillation (TA-OPD): To solve task interference, the authors first train specialized "teacher" models for prediction and refinement. They then consolidate these capabilities into a single "student" model by matching the teachers' "velocities" (predicted directions in the diffusion process) on trajectories actually visited by the student.
Figure 1. SCALEWORLD uses one shared backbone for streaming world prediction and high-resolution refinement. Each completed low-resolution chunk conditions its high-resolution counterpart, while separate causal states retain a long clean prediction history and a short noisy-forward refinement history. Matched crops highlight the recovered structures and textures.

Datasets

The model was trained and evaluated using:

  • Training: Camera-annotated navigation videos for the prediction task and natural high-resolution videos with synthetic degradations for the refinement task.
  • Evaluation: The authors primarily use WBench, a comprehensive benchmark for interactive video world models that measures dimensions like video quality, control adherence, and physical consistency.

Key Insights and Ideas

  • Weight Sharing is a Free Lunch (Almost): Prediction and refinement tasks are similar enough that they can share the same "visual brain." The conflict isn't in the knowledge, but in the optimization path, which TA-OPD successfully navigates.
  • Noisy-Forward History: One of the biggest "aha" moments is that you don't need a perfectly clean high-res history to generate high-res details. Reusing noisy KV states from the previous step's generation is sufficient for temporal continuity and provides a 2x speedup for the refinement stage.
  • Asymmetric Scaling: By keeping the expensive HR history window short and the cheaper LR history window long, the model balances the need for long-term "world" memory with the high cost of "detail" memory.
Figure 2. Overview of SCALEWORLD. Top: Joint prediction and refinement training precedes causal adaptation, few-step specialist training, and TA-OPD. Center: One shared backbone completes each camera-controlled LR prediction and then refines it using the upsampled latent as a condition. WM maintains longer clean history; refinement retains shorter noisy-forward history. HR outputs do not enter the WM history. Bottom: During consolidation, frozen specialists supervise the shared student through local velocity matching, with matched current queries and network-specific KV histories.

Key Results

  • Efficiency: Compared to a standard two-model deployment, ScaleWorld reduces peak GPU memory from 45.4 GiB to 35.8 GiB (~21% saving).
  • Throughput: The "noisy-forward" memory construction enables an end-to-end generation speed of 11.94 FPS, a significant improvement over the 9.41 FPS achieved with standard "clean" history methods.
  • Quality: ScaleWorld achieved an overall WBench score of 81.60, outperforming existing state-of-the-art models like HY-World 1.5 (79.15) and Matrix-Game 3.5 (74.76).
  • Detail Preservation: Qualitative results show that the model effectively recovers fine-grained textures (like waterfront structures and boat details) that are lost in the native low-resolution prediction.
Qualitative comparison showing how ScaleWorld's refinement recovers fine-grained structures and textures.

Limitations and Future Work

  • Visual vs. Physical: The model is a visual predictor, not an explicit physical simulator. It may still generate physically implausible motions in complex scenarios.
  • Feedback Loop: Currently, HR outputs do not feed back into the LR prediction history. This prevents the model from using refined details to inform future predictions, which could potentially help with long-term drift.
  • Domain Robustness: While synthetic degradations help, there remains a "residual gap" between training conditions and the actual errors produced by the world model, which can occasionally affect refinement robustness.

Summary updated

Abstract

High-resolution generation enables video world models to render fine-grained scene structures and textures, but directly generating high-resolution video incurs substantial computational and memory costs. We present ScaleWorld, a causal video model that unifies low-resolution world prediction and high-resolution refinement within a single shared backbone. By refining its own predictions, ScaleWorld produces high-resolution world videos without requiring a separate refinement model. For each video chunk, the shared backbone first completes a low-resolution prediction and then performs one-step high-resolution refinement conditioned on that prediction. To support efficient streaming generation, we maintain separate causal states with a long low-resolution history for prediction and a short high-resolution history for refinement. To address task interference during distillation, we separate specialist optimization from capability consolidation and introduce task-asymmetric on-policy distillation (TA-OPD). This procedure transfers prediction and refinement behaviors from specialized teachers into a shared student through local velocity matching under their respective sampling and history policies. {Extensive experiments demonstrate that ScaleWorld achieves high-quality world generation with improved computational and memory efficiency.
BibTeX
@misc{qu2026scaleworld,
  author = {Yansong Qu and Wesker Zheng and Quanjian Song and Zhiyuan Zhang and Xiao Cao and Shengchuan Zhang and Liujuan Cao},
  title = {{ScaleWorld: High-Resolution World Generation through Shared Prediction and Refinement}},
  year = {2026},
  eprint = {2611.26615},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26615},
}
Figure