UCM: Unifying Camera Control and Memory with Time-aware Positional Encoding Warping for World Models

Tianxing Xu, Zixuan Wang, Guangyuan Wang, Li Hu, Zhongyi Zhang, Peng Zhang, Bang Zhang, Song-Hai Zhang

TL;DR

This paper introduces UCM, a novel framework for generating long-term consistent videos with precise camera control. UCM tackles the challenge of maintaining scene coherence when revisiting areas and enabling user-guided camera movements in world models, achieving state-of-the-art results.

What Problem Does This Paper Address?

Generating realistic and interactive virtual environments, often referred to as "world models," is a key goal in AI research. These models have the potential to revolutionize fields like gaming, simulation, and robotics. However, existing methods struggle with two critical issues:

  1. Long-term Content Consistency: When a virtual camera revisits a previously seen part of a scene, the generated content often drifts or becomes inconsistent with the original appearance. This is a major hurdle for creating believable, persistent virtual worlds.
  2. Precise Camera Control: Users want to guide the virtual camera's movement and viewpoint accurately. Current approaches either sacrifice flexibility for explicit 3D reconstruction or rely on implicit methods that limit controllability and consistency.

This paper aims to bridge this gap by unifying camera control and long-term memory within a single framework.

Methods

UCM builds upon a diffusion transformer (DiT) architecture, a powerful model for generating high-fidelity images and videos. The core innovation lies in a time-aware positional encoding warping mechanism. Here's a breakdown:

  • Unified Modeling: Instead of treating camera control and memory separately, UCM integrates them by establishing robust spatio-temporal correspondences between visual tokens.
  • Time-aware Positional Encoding Warping: This technique warps positional encodings of historical and reference frames to align with the target camera's viewpoint. This allows the model to understand how different views relate to each other in 3D space, crucial for both consistency and control.
  • Dual-Stream Diffusion Transformer: To handle the increased computational load from processing multiple frames and their warped encodings, UCM employs an efficient dual-stream DiT. This architecture uses a sparse attention mechanism to reduce computational overhead while maintaining high-fidelity generation.
  • Data Curation Strategy: To overcome the scarcity of large-scale datasets with long-term scene revisits, the authors developed a data augmentation technique. They use point-cloud rendering with perturbed viewpoints to simulate scene revisiting from monocular videos, enabling training on over 500K videos.

Datasets

  • Training: The model is trained on a large dataset of over 500K monocular videos, augmented using a point-cloud-based rendering strategy to simulate scene revisiting. Specific datasets mentioned for training include Miradata, SpatialVID, and Context-as-Memory.
  • Evaluation: UCM is evaluated on diverse datasets, including RealEstate10K, Tanks & Temples, Context-as-Memory, DL3DV, and MiraData. These datasets cover various environments and styles, both realistic and synthetic.

Key Insights and Ideas

  • Spatio-temporal token correspondence is key: The paper argues that both camera control and long-term memory rely on understanding how visual information relates across space and time. UCM's warping mechanism directly addresses this by creating explicit, token-level correspondences.
  • Implicit 3D priors are limiting: Methods that rely solely on learned 3D information from 2D frames struggle with precise control and consistency. UCM's explicit warping offers a more robust solution.
  • Efficient architecture is crucial for scalability: The dual-stream DiT with sparse attention is vital for making the complex spatio-temporal modeling computationally feasible for long-term generation.
  • Synthetic data generation can bridge dataset gaps: The point-cloud rendering approach effectively creates the necessary long-term revisiting data, which is otherwise rare, enabling training on a much larger scale.

Key Results

  • Camera Control: UCM significantly outperforms existing methods in camera controllability, achieving lower rotation and translation errors. For instance, UCM achieves a RotErr of 1.01° and TransErr of 0.11, outperforming methods like VWM (1.54°, 0.10°) and VMem (2.22°, 0.30°).
  • Long-term Memory: UCM demonstrates superior long-term scene consistency, particularly when revisiting scenes. It achieves the best performance across various metrics like FID, FVD, SSIM, and PSNR in both memory initialization and cycle trajectory evaluations. For example, in cycle trajectory evaluation, UCM achieves an FID of 21.78 and FVD of 61.47, significantly better than VWM (61.43, 179.72) and VMem (4.78, 44.05).
  • Visual Quality: UCM generates high-fidelity videos, outperforming competitors in metrics like FID and FVD.
  • Inference Speed: While not the absolute fastest, UCM offers a good balance between performance and speed, achieving 2.40 s/frame inference time, which is competitive with other advanced methods.

Limitations and Future Work

  • Accumulation of minor errors: The paper acknowledges that over long clip-by-clip sequences, minor prediction errors can accumulate, potentially affecting the overall integrity of the simulation.
  • Distinguishing dynamic vs. static elements: The method relies on learned priors to differentiate dynamic objects from static scenes during memory injection. This can sometimes lead to artifacts when dealing with movable objects.
  • Computational overhead of depth estimation: Streaming depth estimation, while crucial for the warping mechanism, adds to the computational cost.
  • Future Work: The authors suggest that efficiently organizing historical information will be important for practical deployment. Further research could focus on improving the handling of dynamic elements and reducing the computational burden of depth estimation.

Summary updated

Abstract

World models based on video generation demonstrate remarkable potential for simulating interactive environments yet suffer from persistent difficulties in two key areas: maintaining long-term content consistency when scenes are revisited and enabling precise camera control from user-specified inputs. Existing methods based on explicit 3D reconstruction often compromise flexibility in unbounded scenarios and struggle to preserve fine-grained structures. Alternative methods rely directly on previously generated frames without establishing explicit spatial correspondence, thereby limiting controllability and consistency. To address these limitations, we present UCM, a novel framework for unified modeling of long-term memory and precise camera control via a time-aware positional encoding warping mechanism. To reduce computational overhead, we design an efficient dual-stream diffusion transformer for high-fidelity generation. Moreover, we introduce a scalable data curation strategy that utilizes point-cloud-based rendering to simulate scene revisiting, enabling training on over 500K monocular videos. Extensive experiments on real-world and synthetic benchmarks demonstrate that UCM significantly outperforms state-of-the-art methods on long-term scene consistency, while achieving precise camera controllability in high-fidelity video generation.
BibTeX
@inproceedings{xu2026ucm,
  author = {Tianxing Xu and Zixuan Wang and Guangyuan Wang and Li Hu and Zhongyi Zhang and Peng Zhang and Bang Zhang and Song-Hai Zhang},
  title = {{UCM: Unifying Camera Control and Memory with Time-aware Positional Encoding Warping for World Models}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2026},
  doi = {10.1145/3799902.3811088},
}

Loading related content...

Figure