Bring Your Dreams to Life: Continual Text-to-Video Customization

Jiahua Dong, Xudong Wang, Wenqi Liang, Zongyan Han, Meng Cao, Duzhen Zhang, Hanbin Zhao, Zhi Han, Salman Khan, Fahad Shahbaz Khan

TL;DR

This paper introduces Continual Text-to-Video Customization (CTVC), a new paradigm that allows text-to-video models to sequentially learn new personalized subjects and motions over time without retraining from scratch. The authors present the Continual Customized Video Diffusion (CCVD) model, which successfully mitigates both the "forgetting" of old concepts and the "neglect" of multiple concepts during video generation.

What Problem Does This Paper Address?

Customized text-to-video generation (CTVG) has made massive strides, allowing users to generate videos of specific subjects (like a personal pet) or specific motion patterns based on text prompts. However, existing methods operate under a major assumption: that the user's library of personalized concepts is static.

In the real world, users continuously encounter new subjects and motions they want to add to their models. If you sequentially fine-tune a standard video diffusion model on new concepts, you run into two severe roadblocks:

  1. Catastrophic Forgetting: The model loses the unique visual traits of previously learned concepts.
  2. Concept Neglect: When asked to generate a video containing multiple previously learned concepts, the model often completely ignores or omits some of them.

While keeping all past training data and retraining the model from scratch could solve this, it is computationally prohibitive and raises significant user privacy concerns. This paper tackles how to learn these concepts sequentially without storing past video data.

Methods

The authors develop the Continual Customized Video Diffusion (CCVD) framework, which splits the problem into specialized training and inference mechanisms:

Architectural overview of the CCVD model, highlighting the CAR module, TCR strategy, and Controllable Conditional Synthesis.

1. Concept-Specific Attribute Retention (CAR) Module (Training)

To prevent the model from overwriting old concepts during sequential fine-tuning, the CAR module introduces two innovations:

  • Layer-Specific Concept Tokens: Instead of using a uniform text prompt across the entire network, CCVD initializes unique text tokens optimized for each layer of the video diffusion model.
  • Concept Orthogonal Loss ($L_{COL}$): This loss function forces the low-rank adaptation (LoRA) parameter subspaces of new concepts to be mathematically orthogonal to previous ones, ensuring that learning a new concept doesn't degrade the weights of older ones.

2. Task-Aware Concept Aggregation (TCR) Strategy (Inference)

At test time, if a user wants to combine multiple concepts, the TCR strategy dynamically measures the semantic relationship between the target prompt and the stored concept learners. It then calculates a weighted combination of the subject and motion adapters to accurately render the video.

3. Controllable Conditional Synthesis (Inference)

To combat concept neglect in multi-concept videos, the model utilizes user-provided bounding boxes alongside two key components:

  • Layer-Specific Region Attention: This restricts the cross-attention of specific concept tokens strictly to their designated spatial regions.
  • Attention-Guided Noise Estimation: The model calculates the maximum attention value for each region. If a concept has a low attention score (meaning the model is about to neglect it), the system dynamically assigns a larger weight to its noise estimation during the denoising process, forcing the model to render it.

Datasets

Because this is a new problem setting, the authors introduced the Continual Video Customization (CVC) dataset:

  • Composition: 35 sequential text-to-video customization tasks.
  • Content: 25 personalized subjects (9 pets, 5 artistic styles, 11 objects) and 10 distinct motion patterns.
  • Challenge: It explicitly includes highly semantically similar concepts (e.g., seven different dogs and two different cats) to rigorously test the model's ability to avoid catastrophic forgetting and identity blending.

Key Insights and Ideas

  • Orthogonality is Key to Isolation: By treating concept learning as a subspace optimization problem and enforcing orthogonality, the model can safely isolate fine-grained visual features without needing a memory replay buffer of old videos.
  • Attention as a Feedback Loop for Denoising: Using region attention values as a real-time indicator of "neglect" allows the model to dynamically intervene during the iterative denoising process. If the model is failing to generate an object in a bounding box, boosting that region's noise estimation forces the concept into existence.

Key Results

The authors evaluated CCVD against several state-of-the-art continual learning and diffusion baselines (including EWC, LWF, CLoRA, and L2DM) using both DreamVideo and Wan 2.1 as underlying backbones.

Qualitative comparison of CCVD against L2DM across DreamVideo and Wan 2.1 backbones.
  • Quantitative Performance: On the DreamVideo backbone, CCVD achieved the highest text-to-video alignment scores (CLIPI of 64.3) and dramatically lower forgetting metrics (F-CLIPI of 2.3 compared to Finetuning's 14.5 and L2DM's 3.9).
  • Multi-Concept Generation: As seen in the qualitative results, when baselines were tasked with generating multiple concepts simultaneously, they frequently omitted subjects or mixed up styles. CCVD consistently preserved distinct identities and successfully rendered complex multi-concept scenes.
Multi-concept video customization results showing CCVD successfully rendering multiple distinct subjects and motions simultaneously.

Limitations and Future Work

The authors note two primary societal and technical limitations:

  • Copyright Risks: Since the model makes it easy to continuously customize video generation with just a few images, users could potentially fine-tune models on proprietary or copyrighted materials without authorization.
  • Privacy Concerns: In practical applications, continuously storing user-specific concept adapters and preferences requires robust personal data protection workflows.

Summary updated

Abstract

Customized text-to-video generation (CTVG) has recently witnessed great progress in generating tailored videos from user-specific text. However, most CTVG methods assume that personalized concepts remain static and do not expand incrementally over time. Additionally, they struggle with forgetting and concept neglect when continuously learning new concepts, including subjects and motions. To resolve the above challenges, we develop a novel Continual Customized Video Diffusion (CCVD) model, which can continuously learn new concepts to generate videos across various text-to-video generation tasks by tackling forgetting and concept neglect. To address catastrophic forgetting, we introduce a concept-specific attribute retention module and a task-aware concept aggregation strategy. They can capture the unique characteristics and identities of old concepts during training, while combining all subject and motion adapters of old concepts based on their relevance during testing. Besides, to tackle concept neglect, we develop a controllable conditional synthesis to enhance regional features and align video contexts with user conditions, by incorporating layer-specific region attention-guided noise estimation. Extensive experimental comparisons demonstrate that our CCVD outperforms existing CTVG baselines on both the DreamVideo and Wan 2.1 backbones. The code is available at https://github.com/JiahuaDong/CCVD.
BibTeX
@inproceedings{dong2026bring,
  author = {Jiahua Dong and Xudong Wang and Wenqi Liang and Zongyan Han and Meng Cao and Duzhen Zhang and Hanbin Zhao and Zhi Han and Salman Khan and Fahad Shahbaz Khan},
  title = {{Bring Your Dreams to Life: Continual Text-to-Video Customization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i5.37361},
}
Figure