CamCtrl3D: Single-Image Scene Exploration With Precise 3D Camera Control

Stefan Popov, Amit Raj, Yuanzhen Li, Michael Krainin, William T. Freeman, Michael Rubinstein

Introduction

The ultimate goal of view synthesis is to allow users to "walk into" their own photographs. While recent video diffusion models can generate motion, they often lack precise control. If a user specifies a camera trajectory—such as moving two meters forward and orbiting 30 degrees—most models struggle to maintain geometric consistency, resulting in "melting" objects or flickering textures. CamCtrl3D, a new generative model from Google DeepMind, addresses these issues by transforming a single static image into a high-fidelity "fly-through" video based on a precise 3D camera trajectory.

Methodology

CamCtrl3D builds upon Stable Video Diffusion (SVD) and introduces a unified framework that conditions the generation process on four distinct signals:

  1. Raw Camera Extrinsics: 4x3 camera matrices are integrated directly into the model's temporal layers.
  2. Camera Rays: Images encoding the origin and direction of every pixel's ray in 3D space provide spatial context.
  3. Initial Image Re-projection: Using a depth estimation model (ZoeDepth), the first image is converted into a 3D point cloud. This cloud is then "rendered" from the perspective of every future camera pose in the trajectory, creating a "guide" video that indicates where original pixels should appear.
  4. 2D-3D Transformers: Novel blocks lift 2D image features into a global 3D voxel grid using ray-traced attention. This allows the model to reason about the scene in 3D before projecting features back to 2D to guide video generation.
Generated frame at the start of a camera trajectory (#0).

These conditions are integrated using a ControlNet-style architecture, allowing the base video model to remain frozen while the new control layers are trained.

Data and Metric Calibration

The researchers utilized ScanNet, RealEstate10K, and DL3DV for training. A key contribution was calibrating these datasets to a consistent metric scale. Since standard Structure-from-Motion (SfM) poses have arbitrary scales, the authors used depth estimation to normalize all camera trajectories to meters. This ensures the model learns a predictable relationship between camera movement and visual change, a prerequisite for precise control.

Key Results

CamCtrl3D sets a new benchmark for controlled video generation in terms of both efficiency and quality:

  • Efficiency: It outperformed the leading competitor, 4DiM, while using approximately 25x less posed training data (10K videos vs. 250K+).
  • Visual Quality: On the RealEstate10K benchmark, it achieved a 2.7x lower Fréchet Video Distance (FVD) compared to 4DiM.
  • Precision: Compared to MotionCtrl, CamCtrl3D showed an 8x reduction in FVD and a 4 dB gain in PSNR, indicating significantly higher fidelity to the original scene's details during camera movement.
Generated frame later in the trajectory (#24), demonstrating the model's ability to maintain geometric consistency and detail during a "fly-through."

Limitations and Future Work

Currently, CamCtrl3D is optimized for static scenes like houses and landmarks, reflecting its training data. While it can occasionally animate elements like water or clouds due to its SVD priors, it is not yet designed for dynamic actors. Additionally, extremely long trajectories can lead to a loss of coherence as the model is forced to "guess" entirely new environments. Future work will focus on fine-tuning the model on dynamic datasets to combine precise camera control with moving objects.

Summary updated

Abstract

We propose a method for generating fly-through videos of a scene, from a single image and a given camera trajectory. We build upon an image-to-video latent diffusion model. We condition its UNet denoiser on the camera trajectory, using four techniques. (1) We condition UNet's temporal blocks on raw camera extrinsics, similar to MotionCtrl. (2) We use images containing camera ray parameters, similar to CameraCtrl. (3) We re-project the initial image to subsequent frames and condition on the resulting video. (4) We introduce a global 3D representation using 2D$\Leftrightarrow$3D transformers, which implicitly conditions on the camera poses. We combine all conditions in a ContolNet-style architecture. We then propose a metric that evaluates overall video quality and the ability to preserve details with view changes, which we use to analyze the trade-offs of individual and combined conditions. Finally, we identify an optimal combination of conditions. We calibrate camera positions in our datasets for scale consistency across scenes, and we train our scene exploration model, CameraCtrl3D, demonstrating state-of-the-art results. Example videos generated by CameraCtrl3D are available at https://camctrl3d.github.io/
BibTeX
@inproceedings{popov2025camctrl,
  author = {Stefan Popov and Amit Raj and Yuanzhen Li and Michael Krainin and William T. Freeman and Michael Rubinstein},
  title = {{CamCtrl3D: Single-Image Scene Exploration With Precise 3D Camera Control}},
  booktitle = {3DV},
  year = {2025},
  url = {https://openreview.net/forum?id=sRBNwH2Qal},
}
Figure