Introduction
Modern image-to-video (I2V) models often struggle with "precision control," particularly when animating specific parts of an object or handling disocclusions—the pixels revealed behind a moving object. Current models typically rely on stochastic inpainting to "guess" what lies behind, often leading to temporal inconsistency, "ghosting," or a loss of object identity. This paper introduces a training-free pipeline designed to give users deterministic control over both motion and the content revealed during disocclusion.
Methodology
The proposed framework decouples motion from appearance using a two-pass approach centered on a Proxy Dynamic Graph (PDG).
- Motion Definition: Users construct a PDG over the input image. Nodes represent segmented object parts (lifted into 3D via depth estimation), while edges define articulated movements like rotation or translation.
- Initial Generation: A "Diffusion-as-Shader" (DaS) model uses the PDG's tracking video to generate a base animation.
- User-Guided Reveal: To handle disocclusions precisely, users can edit the final frame of the base video (e.g., using generative fill) to specify exactly what should be revealed.
- Latent-Space Compositing: Instead of simple pixel-level pasting, the system blends the latent features of the original image and the user's edit. By injecting these features during the diffusion denoising process, the model harmonizes the new content with the scene's lighting, shadows, and reflections.
Evaluation and Results
To evaluate this new task, the authors established a benchmark of 50 samples focusing on complex articulation and specific "reveals."
The method demonstrates significant improvements over existing state-of-the-art models:
- Motion Accuracy: Quantitative tests using optical flow showed the proposed method achieving a score of 0.72, vastly outperforming baselines like Veo3.1 (0.02), which rely solely on text prompts.
- User Preference: In a study with 32 participants, the method was overwhelmingly preferred for its adherence to motion instructions and realism in disoccluded areas compared to models like Puppet-Master and DragAnything.
Key Insights
The core innovation lies in the decoupling of motion and appearance. By using the PDG as a deterministic skeleton and performing compositing in the latent space rather than the pixel space, the model ensures that new content obeys physical laws like shading while maintaining the structural integrity of the moving objects.
Conclusion
While the current system excels at rigid and semi-rigid deformations, future work aims to address highly fluid dynamics (e.g., water or cloth). Nonetheless, this framework provides a predictable, user-centric workflow for high-quality, articulated video generation that goes "beyond the visible" pixels of the source image.
Summary updated