CamDirector: Towards Long-Term Coherent Video Trajectory Editing

Zhihao Shi, Kejia Yin, Weilin Wan, Yuhongze Zhou, Yuanhao Yu, Xinxin Zuo, Qiang Sun, Juwei Lu

TL;DR

CamDirector is a novel video trajectory editing framework that allows users to smoothly change the camera path of an existing video while maintaining flawless long-term consistency. By decoupling the scene into dynamic objects and a global static "world cache" point cloud, it synthesizes source-aligned, cinematic videos using significantly fewer parameters than current state-of-the-art models.

What Problem Does This Paper Address?

Video Trajectory Editing (VTE) is the art of taking a casual, potentially shaky video and rendering it along a new, professionally styled camera trajectory (like a smooth pan or an orbiting shot). While highly desirable for content creators, VTE suffers from two major challenges:

  1. Source Alignment: Ensuring that the newly generated viewpoints strictly honor the original scene content without introducing spatial mismatches.
  2. Self-Consistency: Preventing visual drift, flickering, or artifacts in newly inpainted (previously unseen) regions over long video sequences.

Existing methods fall short because they either inject camera poses through limited-capacity embedding layers (lacking precise control) or warp frames individually. Frame-by-frame warping relies on implicit attention mechanisms to maintain consistency, which breaks down on long videos due to memory constraints that force chunk-by-chunk processing.

Methods

CamDirector solves these issues by introducing a two-pronged approach: a Hybrid Warping Scheme and a History-Guided Autoregressive Generation model.

Overview of the CamDirector framework, showing the hybrid warping scheme (left) and the CCDM architecture (right).

1. Hybrid Warping Scheme

Instead of treating the entire frame uniformly, CamDirector explicitly separates the scene into dynamic and static regions:

  • Dynamic regions are warped frame-by-frame to preserve motion fidelity.
  • Static regions are progressively aggregated across all source frames into a unified, lightweight 3D point cloud called the world cache.

By rendering this global world cache to the target camera poses and fusing it with the warped dynamic elements, the model creates highly complete and globally consistent "coarse frames" to guide the generation process.

2. History-Guided Autoregressive Generation

To generate long videos without quality degradation, the authors design a Coarse-Video-Controlled Diffusion Model (CCDM) based on the pre-trained Wan-T2V-1.3B architecture. The video is generated in short, sequential segments where the cleaner, previously generated "history" segment directly guides the denoising of the current segment.

Crucially, as new segments are generated, the newly inpainted static regions are fused back into the global world cache. This progressive update ensures that future segments remain perfectly aligned with earlier ones.

Datasets

  • Dynamic Multi-view Dataset: Used for primary training, consisting of roughly 13.6K dynamic scenes. The authors applied epipolar constraints to filter and correct inaccurate depth maps from this dataset, resulting in 9.5K high-quality training scenes.
  • iPhone Dataset: A standard, older benchmark limited to just 5 usable test scenes with narrow fields of view and simple camera motions.
  • iPhone-PTZ (New Benchmark): To rigorously test VTE, the authors introduced this new dataset featuring 10 diverse scenes. It covers complex cinematic camera movements (dolly, pan, orbit, rotation) captured using an iPhone 14 Plus paired with a DJI Osmo Mobile 7P PTZ gimbal, offering much larger trajectory variations and wider fields of view.

Key Insights and Ideas

  • Explicit 3D Memory Beats Implicit Attention: Relying on a diffusion model's internal attention mechanism to remember what happened 50 frames ago is computationally prohibitive. Storing the background explicitly in a 3D world cache bypasses memory limits and guarantees structural permanence.
  • Progressive Cache Fusing Prevents Drift: Standard autoregressive video generation models only pass pixel or latent history forward. By updating the underlying 3D geometry (the world cache) with newly inpainted regions, CamDirector ensures that if the camera loops back to a previously inpainted area, it remains perfectly consistent.
  • Data Cleaning is Crucial: The authors observed that standard depth estimation foundation models often produce poor warping results due to frame-to-frame depth inconsistencies. Implementing a rigorous epipolar filtering step during data preparation was essential to achieving clean synthesis.
Visual comparison showing how corrected depths yield superior, source-aligned warping results compared to standard VGGT depth estimation.

Key Results

CamDirector achieves state-of-the-art performance across both standard and newly introduced benchmarks while being highly parameter-efficient:

  • Efficiency: At 2.0B parameters, CamDirector outperforms significantly larger models, including TrajectoryCrafter (5.3B) and Gen3C (6.7B).
  • Quantitative Performance: On the challenging iPhone-PTZ benchmark for full-length videos, CamDirector achieved a PSNR of 13.99 and an LPIPS of 0.4752, noticeably outperforming Gen3C (13.27 PSNR / 0.5497 LPIPS).
  • Qualitative Consistency: As shown in visual comparisons, the hybrid warping scheme produces far more complete and source-aligned coarse frames than traditional per-frame warping methods, effectively eliminating the blurriness and severe visual artifacts seen in competing approaches.
Visual comparison demonstrating that hybrid warping (b) produces more complete and source-aligned coarse frames than per-frame warping (a).

Limitations and Future Work

The authors note that generated frames can occasionally appear over-smoothed, particularly in regions with intricate or complex textures. This limitation stems from the synthetic training dataset, which features relatively coarse rendered textures.

To address this, future work will focus on incorporating real-world static multi-view datasets into the training pipeline and leveraging advanced computer graphics techniques to build higher-fidelity synthetic datasets with richer, more vivid textural details.

Summary updated

Abstract

Video (camera) trajectory editing aims to synthesize new videos that follow user-defined camera paths while preserving scene content and plausibly inpainting previously unseen regions, upgrading amateur footage into professionally styled videos. Existing VTE methods struggle with precise camera control and long-range consistency because they either inject target poses through a limited-capacity embedding or rely on single-frame warping with only implicit cross-frame aggregation in video diffusion models. To address these issues, we introduce a new VTE framework that 1) explicitly aggregates information across the entire source video via a hybrid warping scheme. Specifically, static regions are progressively fused into a world cache then rendered to target camera poses, while dynamic regions are directly warped; their fusion yields globally consistent coarse frames that guide refinement. 2) processes video segments jointly with their history via a history-guided autoregressive diffusion model, while the world cache is incrementally updated to reinforce already inpainted content, enabling long-term temporal coherence. Finally, we present iPhone-PTZ, a new VTE benchmark with diverse camera motions and large trajectory variations, and achieve state-of-the-art performance with fewer parameters.
BibTeX
@inproceedings{shi2026camdirector,
  author = {Zhihao Shi and Kejia Yin and Weilin Wan and Yuhongze Zhou and Yuanhao Yu and Xinxin Zuo and Qiang Sun and Juwei Lu},
  title = {{CamDirector: Towards Long-Term Coherent Video Trajectory Editing}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=23p42VfqJj},
}
Figure