LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation

Ruiqi Wu, Liangyu Chen, Tong Yang, Chunle Guo, Chongyi Li, Xiangyu ZhangarXiv 2023 · DOI · arXiv

Introduction

Generating high-quality video from text descriptions remains one of the most significant challenges in generative AI. While text-to-image (T2I) models have reached a high level of maturity, text-to-video (T2V) models typically face a trade-off between two extremes. Large-scale models require immense computational resources and millions of video-text pairs, while one-shot methods often "overfit" to the specific content of a single template video, limiting their creative flexibility and trapping the output in the original video's layout.

LAMP (Learn A Motion Pattern) introduces a middle-ground approach. It is a few-shot tuning framework designed to learn generalized motion patterns—such as the specific gait of a running horse or the fluid dynamics of a waterfall—from a small set of 8 to 16 videos. By decoupling the generation of static content from the prediction of motion, LAMP enables the creation of temporally consistent videos that generalize across diverse styles and prompts.

Methodology

The fundamental insight behind LAMP is that the first frame of a video contains the majority of the semantic and structural information. By treating the first frame as a fixed anchor, the model can focus its learning capacity entirely on the temporal evolution of the scene.

First-Frame-Conditioned Pipeline

LAMP utilizes a heterogeneous model architecture to maximize quality. It employs the high-resolution SD-XL model to generate a detailed, high-quality first frame based on the user's prompt. This frame then serves as the condition for a modified, more efficient Stable Diffusion v1.4 model, which is tasked with predicting the subsequent frames. This separation of concerns allows the system to leverage best-in-class image generation while specializing the motion learning on a lighter-weight backbone.

Temporal-Spatial Motion Learning

To adapt the 2D T2I model for video, the authors "inflate" the standard convolutional layers. The architecture introduces:

  • 1D Temporal Convolutions: These layers capture temporal dependencies. Specifically, they are designed to look at the previous two frames to predict the next, mimicking classical video prediction tasks.
  • Spatial Compensation Branch: A small 2D spatial branch with a Sigmoid gate is added to compensate for spatial features that might be lost or distorted by 1D temporal kernels.
  • Zero-Initialization: All newly added temporal layers are initialized to zero. This ensures that the model begins training with the same output as the original T2I model, preserving the base model's generative power while gradually learning to incorporate motion.

Inference and Stability

During the inference phase, LAMP employs Shared-Noise Sampling. By using a consistent base noise across all frames, the model significantly reduces background drifting and flickering, which are common artifacts in zero-shot video generation.

Experiments and Results

The authors evaluated LAMP across eight distinct motion categories, ranging from rigid body movements (helicopters) and fluid dynamics (waterfalls) to complex particle simulations (fireworks) and organic animal/human motions (birds flying, playing guitar).

Performance Metrics

In quantitative evaluations, LAMP outperformed established baselines such as AnimateDiff, T2V-Zero, and Tune-A-Video. The model showed superior performance in three key areas:

  1. Textual Alignment: Measured via CLIP scores, demonstrating that the motion learning does not degrade the model's ability to follow complex prompts.
  2. Temporal Consistency: The videos maintained structural integrity across frames better than one-shot methods.
  3. Generation Diversity: Unlike template-based methods, LAMP can apply a learned motion (e.g., "running") to entirely new subjects (e.g., a "wooden horse" or a "comic style girl") without being tethered to the original training video's specific content.

Efficiency

One of LAMP's most practical advantages is its efficiency. Training can be completed on a single GPU with approximately 15 GB of vRAM, making it accessible for researchers using mid-tier hardware like the RTX 3090 or 4090. Inference is even lighter, requiring only 6 GB of vRAM.

Conclusion

LAMP offers a robust solution for few-shot video generation by focusing on the "how" of movement rather than the "what" of the scene. By decoupling content from motion and utilizing a first-frame-conditioned pipeline, it achieves a level of generalization that was previously difficult to reach without massive datasets. While challenges remain—particularly regarding the unintended coupling of foreground motion and background warping—LAMP represents a significant step toward democratizing high-quality, customizable video generation.

Summary updated

Abstract

With the impressive progress in diffusion-based text-to-image generation, extending such powerful generative ability to text-to-video raises enormous attention. Existing methods either require large-scale text-video pairs and a large number of training resources or learn motions that are precisely aligned with template videos. It is non-trivial to balance a trade-off between the degree of generation freedom and the resource costs for video generation. In our study, we present a few-shot-based tuning framework, LAMP, which enables text-to-image diffusion model Learn A specific Motion Pattern with 8~16 videos on a single GPU. Specifically, we design a first-frame-conditioned pipeline that uses an off-the-shelf text-to-image model for content generation so that our tuned video diffusion model mainly focuses on motion learning. The well-developed text-to-image techniques can provide visually pleasing and diverse content as generation conditions, which highly improves video quality and generation freedom. To capture the features of temporal dimension, we expand the pretrained 2D convolution layers of the T2I model to our novel temporal-spatial motion learning layers and modify the attention blocks to the temporal level. Additionally, we develop an effective inference trick, shared-noise sampling, which can improve the stability of videos with computational costs. Our method can also be flexibly applied to other tasks, e.g. real-world image animation and video editing. Extensive experiments demonstrate that LAMP can effectively learn the motion pattern on limited data and generate high-quality videos. The code and models are available at https://rq-wu.github.io/projects/LAMP.
BibTeX
@misc{wu2023lamp,
  author = {Ruiqi Wu and Liangyu Chen and Tong Yang and Chunle Guo and Chongyi Li and Xiangyu Zhang},
  title = {{LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation}},
  year = {2023},
  eprint = {2310.10769},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.10769},
}
Figure