AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Data

Fu-Yun Wang, Zhaoyang Huang, Weikang Bian, Xiaoyu Shi, Keqiang Sun, Guanglu Song, Yu Liu, Hongsheng Li

Abstract

This paper introduces an effective method for computation-efficient personalized style video generation without requiring access to any personalized video data. It reduces the necessary generation time of similarly sized video diffusion models from 25 seconds to around 1 second while maintaining the same level of performance. The method's effectiveness lies in its dual-level decoupling learning approach: 1) separating the learning of video style from video generation acceleration, which allows for personalized style video generation without any personalized style video data, and 2) separating the acceleration of image generation from the acceleration of video motion generation, enhancing training efficiency and mitigating the negative effects of low-quality video data.

Introduction

Video diffusion models have revolutionized content creation, but they suffer from two major bottlenecks: slow generation speed and style inflexibility. The iterative denoising process is computationally expensive, often taking nearly half a minute for a short clip. Furthermore, while users desire videos in specific artistic styles (e.g., 3D animation, ink painting), high-quality annotated video datasets in these styles are rare. Training on low-quality video data typically results in blurry or flickering outputs. AnimateLCM addresses these issues by proposing a framework that learns "style" from high-quality images and "motion" from generic videos, all while drastically accelerating the inference process.

Methods: Dual-Level Decoupling

The core innovation of AnimateLCM is a dual-level decoupling learning strategy. Instead of training a monolithic model on specific stylized video data, the authors decompose the problem into modular components.

1. Decoupling Style from Acceleration

The authors treat style and speed as separate "weight residuals" that can be plugged into a base Latent Diffusion Model (LDM). Style is captured using LoRA (Low-Rank Adaptation) fine-tuned on high-quality image datasets. Acceleration is achieved through Consistency Distillation, transforming the iterative process into a "Consistency Model" capable of generating results in 1–4 steps. These additive residuals are combined linearly without increasing inference computation time.

2. Decoupling Image and Video Acceleration

To enhance training efficiency, the acceleration process is split. The base LDM is first accelerated for image generation (creating an "Image LCM"). Subsequently, motion modules are trained on readily available, lower-quality video datasets to extend this acceleration to the temporal dimension. This "image-first" approach provides a stable spatial foundation for learning temporal consistency.

The dual-level decoupling learning strategy of AnimateLCM, separating style, motion, and acceleration training.

Key Results

AnimateLCM achieves a massive speedup, generating a 2-second video in approximately 963ms on an NVIDIA A800 GPU, compared to over 23 seconds for traditional models—a 24.47x speedup.

Low-Step Fidelity

While standard models struggle at low step counts (Number of Function Evaluations or NFE), AnimateLCM maintains high visual quality in just 2 to 4 steps. Quantitative benchmarks on UCF-101 show that AnimateLCM significantly outperforms strong baselines like DDIM and DPM-Solver++ in the 1–4 step regime, achieving much lower Fréchet Video Distance (FVD) scores. In VBench evaluations, AnimateLCM achieved a total score of 79.42%, competitive with industry leaders like Pika-1.0 and Gen-2 despite its superior efficiency.

Comparison of AnimateLCM against teacher models (DDIM and DPM) at various sampling steps (NFE).

Conclusion and Future Work

AnimateLCM demonstrates that motion priors from generic videos can be successfully fused with artistic styles from static images to produce high-quality, stylized videos at high speeds. By utilizing weight residuals and skipping memory-intensive Classifier-Free Guidance (CFG), the model achieves state-of-the-art efficiency. Future work will focus on refining the distillation process to achieve high-fidelity "one-step" generation and reducing dependency on the underlying base LDM's capabilities.

Summary updated

BibTeX
@inproceedings{wang2024animatelcm,
  author = {Fu-Yun Wang and Zhaoyang Huang and Weikang Bian and Xiaoyu Shi and Keqiang Sun and Guanglu Song and Yu Liu and Hongsheng Li},
  title = {{AnimateLCM: Computation-Efficient Personalized Style Video Generation without Personalized Video Data}},
  booktitle = {SIGGRAPH Asia Technical Communications},
  year = {2024},
  doi = {10.1145/3681758.3698013},
}
Figure