TL;DR
This paper introduces Continual Text-to-Video Customization (CTVC), a new paradigm that allows text-to-video models to sequentially learn new personalized subjects and motions over time without retraining from scratch. The authors present the Continual Customized Video Diffusion (CCVD) model, which successfully mitigates both the "forgetting" of old concepts and the "neglect" of multiple concepts during video generation.
What Problem Does This Paper Address?
Customized text-to-video generation (CTVG) has made massive strides, allowing users to generate videos of specific subjects (like a personal pet) or specific motion patterns based on text prompts. However, existing methods operate under a major assumption: that the user's library of personalized concepts is static.
In the real world, users continuously encounter new subjects and motions they want to add to their models. If you sequentially fine-tune a standard video diffusion model on new concepts, you run into two severe roadblocks:
- Catastrophic Forgetting: The model loses the unique visual traits of previously learned concepts.
- Concept Neglect: When asked to generate a video containing multiple previously learned concepts, the model often completely ignores or omits some of them.
While keeping all past training data and retraining the model from scratch could solve this, it is computationally prohibitive and raises significant user privacy concerns. This paper tackles how to learn these concepts sequentially without storing past video data.
Methods
The authors develop the Continual Customized Video Diffusion (CCVD) framework, which splits the problem into specialized training and inference mechanisms:
1. Concept-Specific Attribute Retention (CAR) Module (Training)
To prevent the model from overwriting old concepts during sequential fine-tuning, the CAR module introduces two innovations:
- Layer-Specific Concept Tokens: Instead of using a uniform text prompt across the entire network, CCVD initializes unique text tokens optimized for each layer of the video diffusion model.
- Concept Orthogonal Loss ($L_{COL}$): This loss function forces the low-rank adaptation (LoRA) parameter subspaces of new concepts to be mathematically orthogonal to previous ones, ensuring that learning a new concept doesn't degrade the weights of older ones.
2. Task-Aware Concept Aggregation (TCR) Strategy (Inference)
At test time, if a user wants to combine multiple concepts, the TCR strategy dynamically measures the semantic relationship between the target prompt and the stored concept learners. It then calculates a weighted combination of the subject and motion adapters to accurately render the video.
3. Controllable Conditional Synthesis (Inference)
To combat concept neglect in multi-concept videos, the model utilizes user-provided bounding boxes alongside two key components:
- Layer-Specific Region Attention: This restricts the cross-attention of specific concept tokens strictly to their designated spatial regions.
- Attention-Guided Noise Estimation: The model calculates the maximum attention value for each region. If a concept has a low attention score (meaning the model is about to neglect it), the system dynamically assigns a larger weight to its noise estimation during the denoising process, forcing the model to render it.
Datasets
Because this is a new problem setting, the authors introduced the Continual Video Customization (CVC) dataset:
- Composition: 35 sequential text-to-video customization tasks.
- Content: 25 personalized subjects (9 pets, 5 artistic styles, 11 objects) and 10 distinct motion patterns.
- Challenge: It explicitly includes highly semantically similar concepts (e.g., seven different dogs and two different cats) to rigorously test the model's ability to avoid catastrophic forgetting and identity blending.
Key Insights and Ideas
- Orthogonality is Key to Isolation: By treating concept learning as a subspace optimization problem and enforcing orthogonality, the model can safely isolate fine-grained visual features without needing a memory replay buffer of old videos.
- Attention as a Feedback Loop for Denoising: Using region attention values as a real-time indicator of "neglect" allows the model to dynamically intervene during the iterative denoising process. If the model is failing to generate an object in a bounding box, boosting that region's noise estimation forces the concept into existence.
Key Results
The authors evaluated CCVD against several state-of-the-art continual learning and diffusion baselines (including EWC, LWF, CLoRA, and L2DM) using both DreamVideo and Wan 2.1 as underlying backbones.
- Quantitative Performance: On the DreamVideo backbone, CCVD achieved the highest text-to-video alignment scores (CLIPI of 64.3) and dramatically lower forgetting metrics (F-CLIPI of 2.3 compared to Finetuning's 14.5 and L2DM's 3.9).
- Multi-Concept Generation: As seen in the qualitative results, when baselines were tasked with generating multiple concepts simultaneously, they frequently omitted subjects or mixed up styles. CCVD consistently preserved distinct identities and successfully rendered complex multi-concept scenes.
Limitations and Future Work
The authors note two primary societal and technical limitations:
- Copyright Risks: Since the model makes it easy to continuously customize video generation with just a few images, users could potentially fine-tune models on proprietary or copyrighted materials without authorization.
- Privacy Concerns: In practical applications, continuously storing user-specific concept adapters and preferences requires robust personal data protection workflows.
Summary updated