Introduction
Generating high-quality video from text descriptions remains one of the most significant challenges in generative AI. While text-to-image (T2I) models have reached a high level of maturity, text-to-video (T2V) models typically face a trade-off between two extremes. Large-scale models require immense computational resources and millions of video-text pairs, while one-shot methods often "overfit" to the specific content of a single template video, limiting their creative flexibility and trapping the output in the original video's layout.
LAMP (Learn A Motion Pattern) introduces a middle-ground approach. It is a few-shot tuning framework designed to learn generalized motion patterns—such as the specific gait of a running horse or the fluid dynamics of a waterfall—from a small set of 8 to 16 videos. By decoupling the generation of static content from the prediction of motion, LAMP enables the creation of temporally consistent videos that generalize across diverse styles and prompts.
Methodology
The fundamental insight behind LAMP is that the first frame of a video contains the majority of the semantic and structural information. By treating the first frame as a fixed anchor, the model can focus its learning capacity entirely on the temporal evolution of the scene.
First-Frame-Conditioned Pipeline
LAMP utilizes a heterogeneous model architecture to maximize quality. It employs the high-resolution SD-XL model to generate a detailed, high-quality first frame based on the user's prompt. This frame then serves as the condition for a modified, more efficient Stable Diffusion v1.4 model, which is tasked with predicting the subsequent frames. This separation of concerns allows the system to leverage best-in-class image generation while specializing the motion learning on a lighter-weight backbone.
Temporal-Spatial Motion Learning
To adapt the 2D T2I model for video, the authors "inflate" the standard convolutional layers. The architecture introduces:
- 1D Temporal Convolutions: These layers capture temporal dependencies. Specifically, they are designed to look at the previous two frames to predict the next, mimicking classical video prediction tasks.
- Spatial Compensation Branch: A small 2D spatial branch with a Sigmoid gate is added to compensate for spatial features that might be lost or distorted by 1D temporal kernels.
- Zero-Initialization: All newly added temporal layers are initialized to zero. This ensures that the model begins training with the same output as the original T2I model, preserving the base model's generative power while gradually learning to incorporate motion.
Inference and Stability
During the inference phase, LAMP employs Shared-Noise Sampling. By using a consistent base noise across all frames, the model significantly reduces background drifting and flickering, which are common artifacts in zero-shot video generation.
Experiments and Results
The authors evaluated LAMP across eight distinct motion categories, ranging from rigid body movements (helicopters) and fluid dynamics (waterfalls) to complex particle simulations (fireworks) and organic animal/human motions (birds flying, playing guitar).
Performance Metrics
In quantitative evaluations, LAMP outperformed established baselines such as AnimateDiff, T2V-Zero, and Tune-A-Video. The model showed superior performance in three key areas:
- Textual Alignment: Measured via CLIP scores, demonstrating that the motion learning does not degrade the model's ability to follow complex prompts.
- Temporal Consistency: The videos maintained structural integrity across frames better than one-shot methods.
- Generation Diversity: Unlike template-based methods, LAMP can apply a learned motion (e.g., "running") to entirely new subjects (e.g., a "wooden horse" or a "comic style girl") without being tethered to the original training video's specific content.
Efficiency
One of LAMP's most practical advantages is its efficiency. Training can be completed on a single GPU with approximately 15 GB of vRAM, making it accessible for researchers using mid-tier hardware like the RTX 3090 or 4090. Inference is even lighter, requiring only 6 GB of vRAM.
Conclusion
LAMP offers a robust solution for few-shot video generation by focusing on the "how" of movement rather than the "what" of the scene. By decoupling content from motion and utilizing a first-frame-conditioned pipeline, it achieves a level of generalization that was previously difficult to reach without massive datasets. While challenges remain—particularly regarding the unintended coupling of foreground motion and background warping—LAMP represents a significant step toward democratizing high-quality, customizable video generation.
Summary updated