Customizing Motion in Text-to-Video Diffusion Models

Joanna Materzynska, Josef Sivic, Eli Shechtman, Antonio Torralba, Richard Zhang, Bryan C. Russell

Abstract

Recent advancements in text-to-video (T2V) diffusion models have demonstrated remarkable capabilities in generating high-quality video content. However, these models remain constrained by their training data, struggling to produce specific, novel motions not well-represented in their original datasets. We present NewMove, a framework that enables the "personalization" of T2V models with unique motions using only a few example clips. By abstracting motion patterns into a specialized text token (e.g., "V*"), NewMove allows for the generation of these custom movements across diverse subjects, backgrounds, and non-humanoid characters while maintaining temporal consistency and spatial fidelity.

Introduction

While state-of-the-art T2V models can generate a wide array of scenes, they are fundamentally limited when tasked with producing niche or specific dynamic patterns, such as a unique dance or a particular gesture. Existing approaches generally fall into two categories: motion transfer and subject customization. Motion transfer methods often suffer from structural rigidity, making it difficult to alter the subject or context significantly. Conversely, subject customization techniques like DreamBooth excel at learning static object appearances but fail to capture complex dynamic behaviors.

NewMove addresses the challenge of motion abstraction—the ability to learn the essence of a movement from a limited set of videos while disentangling it from the specific appearance of the actors or the background.

Methodology

NewMove fine-tunes a pre-trained T2V diffusion model (specifically the U-Net architecture) using a reconstruction objective. The framework is built upon three core technical innovations:

Selective Parameter Tuning

To balance the learning of new motions with the preservation of general world knowledge, NewMove targets specific model layers. The authors tune the temporal layers, responsible for timing and consistency, alongside the keys and values of spatial cross-attention modules. This approach acknowledges that motion is not purely temporal; it often involves spatial changes, such as disocclusions, as a subject moves.

Video-Based Regularization

To mitigate "catastrophic forgetting"—where the model loses its ability to generate standard videos—NewMove employs a regularization strategy. Training involves a mixture of the target motion and a regularization set of real-world videos. Empirical results suggest that real videos are more effective than synthetic, model-generated ones for maintaining the base model's prior knowledge.

Coarse-Noise Sampling Strategy

Diffusion models typically determine global structure in early denoising steps and fine details in later steps. NewMove utilizes a non-uniform sampling strategy that prioritizes training on early, "coarse" noise levels. This forces the model to focus on the overarching movement pattern rather than the specific textures or colors of the training subjects.

Experimental Results

The framework was evaluated using the Jester dataset for human gestures, WebVid-10M for regularization, and various "in-the-wild" internet clips for qualitative assessment.

Quantitative Performance

On the Jester gesture recognition task, NewMove achieved a 70.6% motion accuracy. This significantly outperforms adapted appearance-based methods like DreamBooth (28.4%) and motion-transfer methods like Tune-A-Video (18.9%), demonstrating its superior ability to capture and replicate specific dynamic patterns.

Qualitative Generalization and User Preference

NewMove demonstrates impressive compositionality. Once a motion is mapped to a token like "V*", it can be combined with novel prompts. For instance, a motion learned from human examples can be applied to a futuristic robot or a group of nurses in a hospital setting.

Figure 1: NewMove demonstrates the ability to learn a specific motion (the "V* dance") from a few input videos (left) and generalize it to diverse subjects and environments (right), including non-humanoid characters and multiple subjects.

In user studies, participants preferred NewMove's outputs over baseline methods in 60-70% of cases, citing superior motion fidelity and temporal smoothness.

Discussion and Limitations

A key insight of this work is that motion abstraction requires updating spatial layers to handle appearance evolution during movement. By focusing on coarse noise levels, the model successfully prioritizes structure over texture.

However, limitations remain. NewMove currently struggles with very fine-grained details, such as intricate finger movements, which are difficult for current T2V architectures to learn from minimal samples. Additionally, the final output quality remains dependent on the capabilities of the underlying pre-trained model. Future work may explore complex motion blending and further reducing the number of required training samples.

Conclusion

NewMove provides a robust solution for personalizing motion in T2V models. By treating motion as a learnable "word" in the model's vocabulary, it opens new avenues for creative video generation, allowing users to define and deploy custom dynamic behaviors with unprecedented flexibility.

Summary updated

BibTeX
@inproceedings{materzynska2024customizing,
  author = {Joanna Materzynska and Josef Sivic and Eli Shechtman and Antonio Torralba and Richard Zhang and Bryan C. Russell},
  title = {{Customizing Motion in Text-to-Video Diffusion Models}},
  booktitle = {ACCV},
  year = {2024},
  doi = {10.1007/978-981-96-0917-8\_7},
}
Figure