TL;DR
This paper introduces I2V-Adapter, a lightweight and plug-and-play module that enables general image-to-video generation by effectively injecting an input image into pretrained text-to-video diffusion models without altering their core weights. It also proposes a novel "Frame Similarity Prior" to enhance video stability and motion control, making it a versatile tool for personalized and controllable video synthesis.
What Problem Does This Paper Address?
Generating coherent and high-quality videos from a single input image and a text prompt (image-to-video or I2V generation) is a challenging task. Existing methods often struggle with either preserving the input image's identity or maintaining temporal consistency. Some approaches modify the fundamental weights of powerful pretrained text-to-video (T2V) models, which can lead to catastrophic forgetting, disrupt the model's prior knowledge, and limit compatibility with popular community-driven tools like personalized models (e.g., DreamBooth, LoRA) and control mechanisms (e.g., ControlNet). Other methods inject image embeddings via cross-attention but often fail to capture fine-grained details and preserve the input image's identity accurately. This leaves a gap for a flexible, efficient, and identity-preserving I2V solution that works seamlessly with existing diffusion model ecosystems.
Methods
The core innovation is the I2V-Adapter, a lightweight module designed to be inserted into existing T2V diffusion models (specifically, the U-Net architecture). Instead of modifying the pretrained model's weights, I2V-Adapter introduces a parallel branch that propagates the unnoised input image to subsequent noised frames. This is achieved through a cross-frame attention mechanism. The adapter uses a trainable query projector and a trainable zero-initialized output projector. Crucially, the attention keys and values for subsequent frames are shared with the first frame and remain unchanged, while only the query weights and output weights are trainable. This minimal parameter count (as low as 24M) significantly reduces training costs and preserves the original model's capabilities.
To further improve video quality, the authors propose a Frame Similarity Prior. This prior leverages the observation that, at low noise levels, consecutive frames in a video are structurally similar. Instead of denoising from pure noise, the model starts denoising from a slightly corrupted version of the first frame. This is implemented using a degradation operation D(x) that combines a masked version of the original image with a blurred version, controlled by a probability p. This prior helps balance motion amplitude and video stability, especially for weaker base T2V models.
Datasets
- Training: WebVid-10M (for initial training) and a self-collected dataset of watermark-free videos.
- Evaluation: MJHQ-30K dataset.
Key Insights and Ideas
- Decoupled Adaptation: The key insight is that you can adapt powerful T2V models for I2V tasks without retraining the entire model. By focusing on injecting the input image information through a separate attention mechanism, the original model's knowledge is preserved.
- Cross-Frame Attention for Identity: Using cross-frame attention to directly query the first frame's features allows for effective propagation of identity and fine-grained details, which is a common failure point for other methods.
- Frame Similarity Prior for Stability: The idea that short video clips have inherent frame similarity, even when corrupted by noise, is a clever way to regularize the generation process and improve stability without complex architectural changes.
- Plug-and-Play Versatility: The minimal parameter count and preservation of original weights make I2V-Adapter highly compatible with existing personalized models (like those fine-tuned with DreamBooth or LoRA) and control tools (like ControlNet), unlocking a wide range of creative applications.
Key Results
- High-Quality Generation: I2V-Adapter produces high-quality and versatile videos, demonstrating strong identity preservation and semantic alignment with text prompts.
- Efficiency: The model requires significantly fewer trainable parameters (down to 24M, ~1% of mainstream models) compared to other approaches, leading to reduced training costs.
- Quantitative Performance: Achieved competitive results across metrics like DoverVQA, CLIPTemp, FlowScore, and WarpingError, often outperforming existing methods. For example, the SDXL version achieved a high CLIPTemp score of 0.9389 and a low WarpingError of 11.8388.
- Qualitative Superiority: Visual comparisons (Figure 8) show that I2V-Adapter generates more coherent and natural videos with better responsiveness to text prompts compared to other state-of-the-art I2V models.
- Versatility: Demonstrated successful integration with personalized T2I models (e.g., CuteYukiMix) and control tools (e.g., ControlNet), producing impressive results in Figure 1 and Figure 6.
Limitations and Future Work
- Video Length and Resolution: The current implementation is constrained to generating 16-frame videos at 512x512 resolution, limiting the creation of longer or higher-resolution content.
- Future Work: The authors plan to address these limitations by incorporating frame interpolation and spatial super-resolution modules to enable the generation of longer and higher-resolution videos.
Summary updated