I2V-Adapter: A General Image-to-Video Adapter for Video Diffusion Models

Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Chongyang Ma, Weiming Hu, Zhengjun Zha, Haibin Huang, Pengfei Wan, Di Zhang

TL;DR

This paper introduces I2V-Adapter, a lightweight and plug-and-play module that enables general image-to-video generation by effectively injecting an input image into pretrained text-to-video diffusion models without altering their core weights. It also proposes a novel "Frame Similarity Prior" to enhance video stability and motion control, making it a versatile tool for personalized and controllable video synthesis.

What Problem Does This Paper Address?

Generating coherent and high-quality videos from a single input image and a text prompt (image-to-video or I2V generation) is a challenging task. Existing methods often struggle with either preserving the input image's identity or maintaining temporal consistency. Some approaches modify the fundamental weights of powerful pretrained text-to-video (T2V) models, which can lead to catastrophic forgetting, disrupt the model's prior knowledge, and limit compatibility with popular community-driven tools like personalized models (e.g., DreamBooth, LoRA) and control mechanisms (e.g., ControlNet). Other methods inject image embeddings via cross-attention but often fail to capture fine-grained details and preserve the input image's identity accurately. This leaves a gap for a flexible, efficient, and identity-preserving I2V solution that works seamlessly with existing diffusion model ecosystems.

Methods

The core innovation is the I2V-Adapter, a lightweight module designed to be inserted into existing T2V diffusion models (specifically, the U-Net architecture). Instead of modifying the pretrained model's weights, I2V-Adapter introduces a parallel branch that propagates the unnoised input image to subsequent noised frames. This is achieved through a cross-frame attention mechanism. The adapter uses a trainable query projector and a trainable zero-initialized output projector. Crucially, the attention keys and values for subsequent frames are shared with the first frame and remain unchanged, while only the query weights and output weights are trainable. This minimal parameter count (as low as 24M) significantly reduces training costs and preserves the original model's capabilities.

To further improve video quality, the authors propose a Frame Similarity Prior. This prior leverages the observation that, at low noise levels, consecutive frames in a video are structurally similar. Instead of denoising from pure noise, the model starts denoising from a slightly corrupted version of the first frame. This is implemented using a degradation operation D(x) that combines a masked version of the original image with a blurred version, controlled by a probability p. This prior helps balance motion amplitude and video stability, especially for weaker base T2V models.

Datasets

  • Training: WebVid-10M (for initial training) and a self-collected dataset of watermark-free videos.
  • Evaluation: MJHQ-30K dataset.

Key Insights and Ideas

  • Decoupled Adaptation: The key insight is that you can adapt powerful T2V models for I2V tasks without retraining the entire model. By focusing on injecting the input image information through a separate attention mechanism, the original model's knowledge is preserved.
  • Cross-Frame Attention for Identity: Using cross-frame attention to directly query the first frame's features allows for effective propagation of identity and fine-grained details, which is a common failure point for other methods.
  • Frame Similarity Prior for Stability: The idea that short video clips have inherent frame similarity, even when corrupted by noise, is a clever way to regularize the generation process and improve stability without complex architectural changes.
  • Plug-and-Play Versatility: The minimal parameter count and preservation of original weights make I2V-Adapter highly compatible with existing personalized models (like those fine-tuned with DreamBooth or LoRA) and control tools (like ControlNet), unlocking a wide range of creative applications.

Key Results

  • High-Quality Generation: I2V-Adapter produces high-quality and versatile videos, demonstrating strong identity preservation and semantic alignment with text prompts.
  • Efficiency: The model requires significantly fewer trainable parameters (down to 24M, ~1% of mainstream models) compared to other approaches, leading to reduced training costs.
  • Quantitative Performance: Achieved competitive results across metrics like DoverVQA, CLIPTemp, FlowScore, and WarpingError, often outperforming existing methods. For example, the SDXL version achieved a high CLIPTemp score of 0.9389 and a low WarpingError of 11.8388.
  • Qualitative Superiority: Visual comparisons (Figure 8) show that I2V-Adapter generates more coherent and natural videos with better responsiveness to text prompts compared to other state-of-the-art I2V models.
  • Versatility: Demonstrated successful integration with personalized T2I models (e.g., CuteYukiMix) and control tools (e.g., ControlNet), producing impressive results in Figure 1 and Figure 6.

Limitations and Future Work

  • Video Length and Resolution: The current implementation is constrained to generating 16-frame videos at 512x512 resolution, limiting the creation of longer or higher-resolution content.
  • Future Work: The authors plan to address these limitations by incorporating frame interpolation and spatial super-resolution modules to enable the generation of longer and higher-resolution videos.

Summary updated

Abstract

Text-guided image-to-video (I2V) generation aims to generate a coherent video that preserves the identity of the input image and semantically aligns with the input prompt. Existing methods typically augment pretrained text-to-video (T2V) models by either concatenating the image with noised video frames channel-wise before being fed into the model or injecting the image embedding produced by pretrained image encoders in cross-attention modules. However, the former approach often necessitates altering the fundamental weights of pretrained T2V models, thus restricting the model's compatibility within the open-source communities and disrupting the model's prior knowledge. Meanwhile, the latter typically fails to preserve the identity of the input image. We present I2V-Adapter to overcome such limitations. I2V-Adapter adeptly propagates the unnoised input image to subsequent noised frames through a cross-frame attention mechanism, maintaining the identity of the input image without any changes to the pretrained T2V model. Notably, I2V-Adapter only introduces a few trainable parameters, significantly alleviating the training cost and also ensures compatibility with existing community-driven personalized models and control tools. Moreover, we propose a novel Frame Similarity Prior to balance the motion amplitude and the stability of generated videos through two adjustable control coefficients. Our experimental results demonstrate that I2V-Adapter is capable of producing high-quality videos. This performance, coupled with its agility and adaptability, represents a substantial advancement in the field of I2V, particularly for personalized and controllable applications.
BibTeX
@inproceedings{guo2024i,
  author = {Xun Guo and Mingwu Zheng and Liang Hou and Yuan Gao and Yufan Deng and Chongyang Ma and Weiming Hu and Zhengjun Zha and Haibin Huang and Pengfei Wan and Di Zhang},
  title = {{I2V-Adapter: A General Image-to-Video Adapter for Video Diffusion Models}},
  booktitle = {SIGGRAPH (Conference Paper Track)},
  year = {2024},
  doi = {10.1145/3641519.3657407},
}
Figure