Introduction
The ultimate goal of view synthesis is to allow users to "walk into" their own photographs. While recent video diffusion models can generate motion, they often lack precise control. If a user specifies a camera trajectory—such as moving two meters forward and orbiting 30 degrees—most models struggle to maintain geometric consistency, resulting in "melting" objects or flickering textures. CamCtrl3D, a new generative model from Google DeepMind, addresses these issues by transforming a single static image into a high-fidelity "fly-through" video based on a precise 3D camera trajectory.
Methodology
CamCtrl3D builds upon Stable Video Diffusion (SVD) and introduces a unified framework that conditions the generation process on four distinct signals:
- Raw Camera Extrinsics: 4x3 camera matrices are integrated directly into the model's temporal layers.
- Camera Rays: Images encoding the origin and direction of every pixel's ray in 3D space provide spatial context.
- Initial Image Re-projection: Using a depth estimation model (ZoeDepth), the first image is converted into a 3D point cloud. This cloud is then "rendered" from the perspective of every future camera pose in the trajectory, creating a "guide" video that indicates where original pixels should appear.
- 2D-3D Transformers: Novel blocks lift 2D image features into a global 3D voxel grid using ray-traced attention. This allows the model to reason about the scene in 3D before projecting features back to 2D to guide video generation.
These conditions are integrated using a ControlNet-style architecture, allowing the base video model to remain frozen while the new control layers are trained.
Data and Metric Calibration
The researchers utilized ScanNet, RealEstate10K, and DL3DV for training. A key contribution was calibrating these datasets to a consistent metric scale. Since standard Structure-from-Motion (SfM) poses have arbitrary scales, the authors used depth estimation to normalize all camera trajectories to meters. This ensures the model learns a predictable relationship between camera movement and visual change, a prerequisite for precise control.
Key Results
CamCtrl3D sets a new benchmark for controlled video generation in terms of both efficiency and quality:
- Efficiency: It outperformed the leading competitor, 4DiM, while using approximately 25x less posed training data (10K videos vs. 250K+).
- Visual Quality: On the RealEstate10K benchmark, it achieved a 2.7x lower Fréchet Video Distance (FVD) compared to 4DiM.
- Precision: Compared to MotionCtrl, CamCtrl3D showed an 8x reduction in FVD and a 4 dB gain in PSNR, indicating significantly higher fidelity to the original scene's details during camera movement.
Limitations and Future Work
Currently, CamCtrl3D is optimized for static scenes like houses and landmarks, reflecting its training data. While it can occasionally animate elements like water or clouds due to its SVD priors, it is not yet designed for dynamic actors. Additionally, extremely long trajectories can lead to a loss of coherence as the model is forced to "guess" entirely new environments. Future work will focus on fine-tuning the model on dynamic datasets to combine precise camera control with moving objects.
Summary updated