VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors

Juil Koo, Paul Guerrero, Chun-Hao Paul Huang, Duygu Ceylan, Minhyuk Sung

TL;DR

VideoHandles is a novel, training-free framework that allows users to edit the 3D position and rotation of objects within a video while maintaining perfect temporal consistency. By warping the intermediate features of a video generative model through a 3D reconstruction of the scene, it produces realistic edits that automatically update complex effects like shadows, reflections, and disocclusions.

What Problem Does This Paper Address?

While we’ve become quite good at editing the composition of 2D images—moving a vase on a table or shifting a car in a driveway—doing the same in video remains a massive challenge.

If you try to move an object in a video frame-by-frame using image-editing tools, the result is usually a flickering mess because the edits aren't "aware" of each other across time. Furthermore, moving an object in 3D space requires the model to understand what was behind the object (disocclusion) and how the lighting, shadows, and reflections should change in the new position. Current video editing methods mostly focus on changing an object's appearance (e.g., turning a cat into a tiger) or the overall camera motion, leaving 3D spatial composition largely unexplored.

Methods

The authors propose VideoHandles, a zero-shot (no training required) approach that treats a video generative model as a powerful prior for how scenes should look. The pipeline works in four main stages:

  1. 3D Reconstruction: The system uses DUST3R to create a point cloud and estimate camera poses from the source video. This gives the model a "ground truth" 3D understanding of the static scene.
  2. 3D-Aware Feature Warping: Instead of warping raw pixels—which often leads to blurry or noisy artifacts—VideoHandles warps the intermediate features (specifically the queries, keys, and values from spatial self-attention layers) of a video model like OpenSora. These features are "lifted" to the 3D point cloud, transformed according to the user's edit, and projected back into each frame.
  3. Energy-Based Guidance: These warped features act as a guide. During the generative denoising process, the model is nudged to ensure its internal features match the target warped features. This forces the object into its new position while allowing the generative model to naturally "fill in" realistic details like new shadows or reflections.
  4. Self-Attention Masking & Null-Text Prediction: To prevent the original object from appearing as a "ghost" in its old spot, the authors use null-text embeddings in the original region. They also use self-attention maps to create dynamic masks that tell the model exactly which parts of the background (like a table surface) need to be updated to accommodate the object's new position.
Figure 2. VideoHandles Architecture. We use the intermediate features Ψsrc of a video generative model to represent the identity of objects in a source video. Given a 3D transformation of an object, we can use a 3D reconstruction of the scene to warp the intermediate features consistently across frames. Guiding the video generator with these warped features Ψtgt gives us a an edited video where the object is transformed, while also maintaining the plausibility of effects like shadows and reflections.

Datasets

  • Fine-tuning: To improve the realism of the base model (OpenSora), the authors lightly fine-tuned it on RealEstate10K, a dataset containing over 70,000 indoor scene videos.
  • Evaluation: The method was tested on 27 generated videos (320x320 resolution, 51 frames) and several real-world captured videos.

Key Insights and Ideas

  • Features > Pixels: Warping in feature space is the "secret sauce." Features are more robust to small errors in 3D reconstruction than raw pixels. Because the generative model "interprets" these features, it can synthesize a plausible environment around the moved object rather than just copy-pasting it.
  • Spatial vs. Temporal Layers: The authors discovered that guiding the model using spatial self-attention features is much more effective for local edits. Guiding via temporal layers tended to cause unwanted global changes across the entire frame.
  • Attention as a Contextual Mask: By looking at where the moved object "pays attention," the system can automatically identify regions that need semantic adjustment—like the area where a new reflection should appear on a wine glass.
Visualization of self-attention masks showing how the model identifies regions requiring adjustment, such as new reflections.

Key Results

VideoHandles significantly outperforms existing baselines that attempt to solve this problem by editing frames individually.

  • Temporal Consistency: In quantitative tests using Frame LPIPS (a metric for video stability), VideoHandles scored 3.71, vastly better than the state-of-the-art image-based baseline, DiffusionHandles, which scored 18.63.
  • User Preference: In a user study, participants preferred VideoHandles 100% of the time for plausibility when compared to DiffusionHandles.
  • Efficiency: Because it processes all frames in a single feed-forward pass rather than frame-by-frame, it reduced editing time from 96 minutes down to just 6 minutes for a 51-frame video.
Qualitative comparison showing VideoHandles (right) successfully moving objects with realistic shadows and no duplication compared to baselines.

Limitations and Future Work

The primary limitation is that VideoHandles currently only works for static scenes (where the camera moves but the objects are stationary). If an object is already moving in the source video, the 3D reconstruction becomes much more complex.

Additionally, the quality of the output is still tethered to the underlying video generative model. As models like CogVideoX or Sora continue to improve, the authors expect VideoHandles to produce even higher-fidelity results. Future work will focus on extending these 3D compositional edits to dynamic scenes with moving subjects.

Summary updated

Abstract

Generative methods for image and video editing use generative models as priors to perform edits despite incomplete information, such as changing the composition of 3D objects shown in a single image. Recent methods have shown promising composition editing results in the image setting, but in the video setting, editing methods have focused on editing object's appearance and motion, or camera motion, and as a result, methods to edit object composition in videos are still missing. We propose \name as a method for editing 3D object compositions in videos of static scenes with camera motion. Our approach allows editing the 3D position of a 3D object across all frames of a video in a temporally consistent manner. This is achieved by lifting intermediate features of a generative model to a 3D reconstruction that is shared between all frames, editing the reconstruction, and projecting the features on the edited reconstruction back to each frame. To the best of our knowledge, this is the first generative approach to edit object compositions in videos. Our approach is simple and training-free, while outperforming state-of-the-art image editing baselines.
BibTeX
@inproceedings{koo2025videohandles,
  author = {Juil Koo and Paul Guerrero and Chun-Hao Paul Huang and Duygu Ceylan and Minhyuk Sung},
  title = {{VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01648},
}
Figure