Bring Your Dreams to Life: Continual Text-to-Video Customization

Jiahua Dong, Xudong Wang, Wenqi Liang, Zongyan Han, Meng Cao, Duzhen Zhang, Hanbin Zhao, Zhi Han, Salman H. Khan, Fahad Shahbaz Khan

Introduction

Customized text-to-video generation (CTVG) has empowered users to create videos featuring personal subjects, such as a specific pet, or unique motion patterns. However, existing frameworks typically operate under a static assumption: the set of concepts to be learned is fixed. In practice, users often wish to expand their library of personalized concepts over time. This introduces the challenge of Continual Text-to-Video Customization (CTVC).

Standard models face two primary hurdles in this setting: catastrophic forgetting, where learning a new concept (e.g., a new cat) degrades the model's memory of previous ones (e.g., a previously learned dog), and concept neglect, where the model fails to render multiple learned concepts simultaneously in complex prompts.

The CCVD Framework

To address these challenges, the Continual Customized Video Diffusion (CCVD) model introduces three core innovations built upon pretrained diffusion architectures:

1. Concept-Specific Attribute Retention (CAR)

To prevent the overwriting of previous knowledge, CCVD utilizes layer-specific concept tokens rather than a single global prompt. A concept orthogonal loss is applied during training, ensuring that weight updates for new concepts are mathematically independent of the subspaces occupied by existing concepts. This keeps different concepts in their own "mathematical lanes" to mitigate forgetting.

2. Task-Aware Concept Aggregation (TCR)

During the inference stage, the model dynamically weights learned adapters based on their relevance to the user's text prompt. This ensures that the model selectively activates only the necessary knowledge for a given scene, rather than blindly averaging all stored concepts.

3. Attention-Guided Noise Estimation

To combat concept neglect, CCVD employs layer-specific region attention guided by user-provided bounding boxes. By monitoring attention scores, the model identifies areas where a concept is being overlooked and applies attention-guided noise estimation, boosting the diffusion weight in those specific regions to force the rendering of the neglected subject.

Multi-concept video generation featuring a custom dog, teddy bear, and cat, demonstrating the model's ability to handle multiple identities simultaneously.

Experimental Evaluation

The authors introduced the Continual Video Customization (CVC) dataset to benchmark this task. The dataset comprises 35 sequential tasks, including 25 personalized subjects (pets, styles, and objects) and 10 motion patterns. Notably, it includes semantically similar subjects to test the model's ability to maintain distinct identities over time.

Key Results

CCVD significantly outperforms existing continual learning and video diffusion baselines like L2DM and CLoRA:

  • Image Alignment (CLIPI): CCVD achieved a score of 64.3, surpassing the best baseline (62.3).
  • Forgetting Rate (F-CLIPI): CCVD demonstrated superior memory retention with an F-CLIPI score of 2.3, compared to 3.9 for L2DM and 14.5 for standard fine-tuning.

Qualitatively, CCVD excels at generating complex scenes that require the simultaneous rendering of multiple custom identities and motions. While baseline models often merge identities or ignore subjects entirely, CCVD maintains high fidelity to all requested concepts.

Conclusion and Ethical Considerations

CCVD represents a significant step forward in making video generation a truly personalized and evolving tool. However, the ability to perfectly retain and reproduce specific styles and subjects raises important questions regarding copyright and privacy. Future research must focus on integrating robust safeguards to protect intellectual property and user data while maintaining the creative flexibility offered by continual learning models.

Summary updated

BibTeX
@inproceedings{dong2026bring,
  author = {Jiahua Dong and Xudong Wang and Wenqi Liang and Zongyan Han and Meng Cao and Duzhen Zhang and Hanbin Zhao and Zhi Han and Salman H. Khan and Fahad Shahbaz Khan},
  title = {{Bring Your Dreams to Life: Continual Text-to-Video Customization}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i5.37361},
}
Figure