Abstract
Personalizing text-to-video (T2V) models to specific subjects or styles typically requires large-scale customized video datasets, which are often unavailable. We present Still-Moving, a novel framework that enables T2V customization using only a few still images. By decoupling spatial identity from temporal motion through a two-stage adapter mechanism, Still-Moving successfully integrates customized text-to-image (T2I) weights into pre-trained video models without compromising motion quality or subject fidelity.
Introduction
Recent advancements in text-to-image (T2I) personalization, such as DreamBooth and StyleDrop, allow users to fine-tune models on specific subjects or artistic styles. However, extending these capabilities to video generation remains a significant challenge. State-of-the-art T2V models are often "inflated" versions of T2I models, incorporating temporal layers to manage motion. Naively injecting customized T2I weights into these models frequently results in visual artifacts, loss of identity, or static outputs. This failure stems from a distribution mismatch: the temporal layers, trained on the original model's features, cannot interpret the "foreign" features of the customized model. Still-Moving addresses this by aligning these distributions without requiring video data of the target subject.
Methodology
Still-Moving employs a two-stage adapter framework designed to treat motion as a controllable "switch."
The Motion Switch Mechanism
The framework first trains lightweight Motion Adapters (LoRA layers) on the temporal attention blocks of a base T2V model. These adapters are trained to produce "frozen videos"—sequences where every frame is identical. This forces the model into a state where it expects static scenes, providing a controlled environment for feature alignment.
Spatial Alignment
In the second stage, customized T2I weights (e.g., a specific character or style) are injected into the T2V model. Spatial Adapters—simple linear projections—are then trained to map the features of the customized model to the distribution expected by the temporal layers. By training these adapters on "frozen videos" generated by the custom T2I model while the Motion Adapters are active, the system learns to preserve spatial identity without the interference of motion.
Inference and Motion Restoration
During inference, the Motion Adapters are removed, while the Spatial Adapters are retained. This "unleashes" the original T2V model's motion priors, now applied to the correctly aligned custom features. Interestingly, the authors discovered that setting the Motion Adapter scale to a negative value can further amplify dynamic intensity.
Experimental Results
The framework was evaluated across 10 customization tasks, including personalized objects (e.g., specific pets or toys) and unique artistic styles.
Qualitative Performance
Still-Moving demonstrates superior ability in maintaining identity fidelity and style adherence compared to baseline methods like naive weight injection or interpolation. For instance, complex styles like "origami" or "melting gold" are preserved throughout the motion sequence without the subject dissolving into generic forms.
Quantitative Evaluation
Using CLIP-based metrics, Still-Moving outperformed baselines in both Subject Fidelity (CLIP-I) and Prompt Alignment (CLIP-T). In personalization tasks, it achieved a CLIP-I score of 0.772, significantly higher than the 0.680 achieved by naive injection. Furthermore, a user study with 31 participants showed an overwhelming preference for Still-Moving in terms of motion quality and identity preservation.
Conclusion
Still-Moving provides a robust solution for customized video generation in the absence of video training data. By aligning feature distributions through a "motion switch" mechanism, it bridges the gap between static T2I personalization and dynamic T2V generation. Future work will explore scaling this framework to larger models and enhancing complex motion control.
Summary updated