Introduction
Animating a single static image into a realistic, temporally coherent video remains a significant challenge in computer vision. While text-to-video (T2V) diffusion models have demonstrated remarkable generative capabilities, adapting them for image-to-video (I2V) tasks often results in a fundamental trade-off. Existing methods frequently struggle to maintain high visual conformity to the source image while simultaneously producing fluid, physically plausible motion. DynamiCrafter addresses this by leveraging the rich motion priors of pre-trained T2V models through a novel dual-stream injection mechanism.
Methodology
DynamiCrafter builds upon the VideoCrafter T2V framework. To bridge the gap between static image preservation and dynamic motion synthesis, the authors propose a dual-stream image injection paradigm.
Dual-Stream Injection
The architecture, illustrated in Figure 1, processes the input image through two distinct paths:
- Text-aligned Context Representation: The input image is encoded via a pre-trained CLIP image encoder. A learnable Query Transformer (P) then maps these features into a space compatible with the T2V model’s cross-attention layers. This stream provides high-level semantic context, guiding the model on what the scene contains to ensure logical motion.
- Visual Detail Guidance (VDG): To preserve fine-grained pixel details, the original image is concatenated directly with the initial Gaussian noise at the input of the denoising U-Net. This provides low-level guidance to maintain the identity and appearance of the source image.
Adaptive Gating
A critical component of the fusion process is Adaptive Gating. A learnable coefficient, $\lambda$, modulates the influence of the image context across different layers of the U-Net. As shown in Figure 3, the model naturally learns to prioritize image information in the "appearance" layers (the input and output stages of the U-Net) while allowing greater generative freedom in the "structural" middle layers. This spatial-temporal balance allows the model to animate objects without distorting their fundamental visual characteristics.
Experiments and Results
The model was trained on the WebVid-10M dataset. Quantitative evaluations on UCF-101 and MSR-VTT in zero-shot settings demonstrate that DynamiCrafter significantly outperforms existing open-source baselines. For instance, on the MSR-VTT benchmark, DynamiCrafter achieved a Fréchet Video Distance (FVD) of 234.66, a substantial improvement over VideoComposer's 377.29.
Qualitative user studies further validate these results, with participants preferring DynamiCrafter's motion quality and temporal coherence over 38% of the time—nearly double the preference rate of its closest rivals. The model also exhibits versatility in downstream tasks such as storytelling, looping video generation, and generative frame interpolation.
Discussion and Limitations
While DynamiCrafter represents a significant step forward in open-domain image animation, several limitations remain. The current implementation is restricted to a 256x256 resolution and 16-frame clips. Additionally, the frame-by-frame nature of the VAE decoder can occasionally introduce flickering or "breathing" artifacts, particularly in complex textures or human faces. Future work will focus on scaling to higher resolutions and implementing more localized, precise motion control.
Summary updated