Still-Moving: Customized Video Generation without Customized Video Data

Hila Chefer, Shiran Zada, Roni Paiss, Ariel Ephrat, Omer Tov, Michael Rubinstein, Lior Wolf, Tali Dekel, Tomer Michaeli, Inbar Mosseri

Abstract

Personalizing text-to-video (T2V) models to specific subjects or styles typically requires large-scale customized video datasets, which are often unavailable. We present Still-Moving, a novel framework that enables T2V customization using only a few still images. By decoupling spatial identity from temporal motion through a two-stage adapter mechanism, Still-Moving successfully integrates customized text-to-image (T2I) weights into pre-trained video models without compromising motion quality or subject fidelity.

Introduction

Recent advancements in text-to-image (T2I) personalization, such as DreamBooth and StyleDrop, allow users to fine-tune models on specific subjects or artistic styles. However, extending these capabilities to video generation remains a significant challenge. State-of-the-art T2V models are often "inflated" versions of T2I models, incorporating temporal layers to manage motion. Naively injecting customized T2I weights into these models frequently results in visual artifacts, loss of identity, or static outputs. This failure stems from a distribution mismatch: the temporal layers, trained on the original model's features, cannot interpret the "foreign" features of the customized model. Still-Moving addresses this by aligning these distributions without requiring video data of the target subject.

Methodology

Still-Moving employs a two-stage adapter framework designed to treat motion as a controllable "switch."

The Motion Switch Mechanism

The framework first trains lightweight Motion Adapters (LoRA layers) on the temporal attention blocks of a base T2V model. These adapters are trained to produce "frozen videos"—sequences where every frame is identical. This forces the model into a state where it expects static scenes, providing a controlled environment for feature alignment.

Spatial Alignment

In the second stage, customized T2I weights (e.g., a specific character or style) are injected into the T2V model. Spatial Adapters—simple linear projections—are then trained to map the features of the customized model to the distribution expected by the temporal layers. By training these adapters on "frozen videos" generated by the custom T2I model while the Motion Adapters are active, the system learns to preserve spatial identity without the interference of motion.

High-fidelity customization of a specific character (pink cat) using Still-Moving.

Inference and Motion Restoration

During inference, the Motion Adapters are removed, while the Spatial Adapters are retained. This "unleashes" the original T2V model's motion priors, now applied to the correctly aligned custom features. Interestingly, the authors discovered that setting the Motion Adapter scale to a negative value can further amplify dynamic intensity.

Experimental Results

The framework was evaluated across 10 customization tasks, including personalized objects (e.g., specific pets or toys) and unique artistic styles.

Qualitative Performance

Still-Moving demonstrates superior ability in maintaining identity fidelity and style adherence compared to baseline methods like naive weight injection or interpolation. For instance, complex styles like "origami" or "melting gold" are preserved throughout the motion sequence without the subject dissolving into generic forms.

Origami bird motion sequence demonstrating style preservation and fluid movement.

Quantitative Evaluation

Using CLIP-based metrics, Still-Moving outperformed baselines in both Subject Fidelity (CLIP-I) and Prompt Alignment (CLIP-T). In personalization tasks, it achieved a CLIP-I score of 0.772, significantly higher than the 0.680 achieved by naive injection. Furthermore, a user study with 31 participants showed an overwhelming preference for Still-Moving in terms of motion quality and identity preservation.

Conclusion

Still-Moving provides a robust solution for customized video generation in the absence of video training data. By aligning feature distributions through a "motion switch" mechanism, it bridges the gap between static T2I personalization and dynamic T2V generation. Future work will explore scaling this framework to larger models and enhancing complex motion control.

Summary updated

Abstract

Customizing text-to-image (T2I) models has seen tremendous progress recently, particularly in areas such as personalization, stylization, and conditional generation. However, expanding this progress to video generation is still in its infancy, primarily due to the lack of customized video data. In this work, we introduce Still-Moving, a novel generic framework for customizing a text-to-video (T2V) model, without requiring any customized video data. The framework applies to the prominent T2V design where the video model is built over a T2I model (e.g., via inflation). We assume access to a customized version of the T2I model, trained only on still image data (e.g., using DreamBooth). Naively plugging in the weights of the customized T2I model into the T2V model often leads to significant artifacts or insufficient adherence to the customization data. To overcome this issue, we train lightweight Spatial Adapters that adjust the features produced by the injected T2I layers. Importantly, our adapters are trained on "frozen videos" (i.e., repeated images), constructed from image samples generated by the customized T2I model. This training is facilitated by a novel Motion Adapter module, which allows us to train on such static videos while preserving the motion prior of the video model. At test time, we remove the Motion Adapter modules and leave in only the trained Spatial Adapters. This restores the motion prior of the T2V model while adhering to the spatial prior of the customized T2I model. We demonstrate the effectiveness of our approach on diverse tasks including personalized, stylized, and conditional generation. In all evaluated scenarios, our method seamlessly integrates the spatial prior of the customized T2I model with a motion prior supplied by the T2V model.
BibTeX
@article{chefer2024still,
  author = {Hila Chefer and Shiran Zada and Roni Paiss and Ariel Ephrat and Omer Tov and Michael Rubinstein and Lior Wolf and Tali Dekel and Tomer Michaeli and Inbar Mosseri},
  title = {{Still-Moving: Customized Video Generation without Customized Video Data}},
  journal = {ACM Trans. Graphics},
  year = {2024},
  doi = {10.1145/3687945},
}

Loading related content...

Figure