TL;DR
ID-Crafter introduces a novel framework for multi-subject video generation that effectively preserves individual identities and ensures semantic coherence. By combining a hierarchical identity-preserving attention mechanism, dynamic guidance from a Vision-Language Model (VLM), and an online reinforcement learning (RL) fine-tuning phase, it sets a new state-of-the-art on multi-subject video benchmarks.
What Problem Does This Paper Address?
While single-subject text-to-video generation has seen massive progress, scaling this to multi-subject scenarios remains a major hurdle. When a prompt describes multiple distinct entities interacting (e.g., a specific person holding a custom product), existing diffusion models suffer from "identity leakage" or "bleeding"—where features of one subject accidentally blend into another, or the model fails to maintain temporal consistency across frames.
The core challenge lies in the inherent tension between preserving fine-grained, individual subject attributes and generating a fluid, cohesive dynamic scene. Current methods struggle to resolve these semantic conflicts, leading to identity degradation or poor prompt alignment.
Methods
ID-Crafter tackles this via three core innovations built on top of a latent video diffusion transformer (specifically, the DiT-based Wan-Video architecture):
- Hierarchical Identity-Preserving Attention: Instead of simply concatenating all tokens, this module operates in three cascaded stages within the DiT blocks:
- Intra-subject attention: Refines features within each subject independently to capture unique details.
- Gated inter-subject attention: Models interactions between different subjects using a novel gated cross-attention mechanism to prevent identity bleed.
- Cross-modal attention: Fuses the refined subject tokens and text tokens with the main video latent tokens.
- VLM-Grounded Semantic Guidance: Rather than using a conventional text encoder alone, ID-Crafter integrates a pretrained Vision-Language Model (Qwen2.5-VL) as a dynamic reasoning engine. The VLM processes both the text prompt and reference images to provide granular cross-modal guidance.
- Online Reinforcement Learning via Flow-GRPO: To directly optimize the trade-off between identity fidelity and video quality, the authors introduce an online RL stage using Groupwise Policy Optimization (GRPO). This bypasses the need for a fragile value network by using groupwise comparisons for advantage estimation, paired with a composite reward function tracking facial similarity (ArcFace), holistic consistency (NexusScore), aesthetics, and physical plausibility (NaturalScore).
Datasets
To overcome the lack of high-quality multi-subject video training data, the authors proposed a comprehensive data curation pipeline combining authentic and synthetic sources:
- OpenS2V-Nexus: A foundational dataset containing 218,230 general videos, 535,259 masked subjects, and 3,098 generated subjects.
- High-Quality Internal Collection: 9,374 pristine videos and 3,000 generated subjects to enhance appearance diversity and mitigate "copy-plate" artifacts.
- Subject Image Dataset: 14,976 static images (2,877 human-centric and 3,458 isolated clothing items) providing rich appearance priors.
- Evaluation: Tested on a held-out set of 180 unique subject-text pairs from the OpenS2V-Nexus benchmark to ensure an unbiased evaluation.
Key Insights and Ideas
- VLMs as Dynamic Guides, Not Just Encoders: Traditional text encoders struggle with spatial layout and complex multi-subject relationships. Bridging a VLM's high-level semantic reasoning directly into the low-level spatial layout control of a DiT prevents semantic conflicts.
- The Power of Online Exploration: Offline RL methods (like DPO) rely on a static preference dataset, which limits coverage. Online GRPO allows the policy to actively explore the generation space and receive immediate feedback from the composite reward model, significantly boosting identity preservation without causing catastrophic forgetting.
- Mitigating Reward Hacking: The authors found that optimizing purely for aesthetics or face similarity can lead to unnatural artifacts. Introducing a VLM-driven
NaturalScore(evaluating physical laws and common sense) acts as a crucial regularizer against reward hacking.
Key Results
- Quantitative Superiority: ID-Crafter significantly outperforms existing open-source models (Phantom, VACE, SkyReels) and matches or beats proprietary baselines (Kling 1.6, Pika 2.1).
- RL Boost: Adding the online RL phase to the 1.3B base model boosted the FaceSim score from 58.12% to 66.10% (a 13.7% relative improvement) and improved overall text-video alignment.
- Qualitative Consistency: ID-Crafter maintains stable representations of all subjects across long sequences without identity drift, accurately reflecting their appearance from the reference images.
Limitations and Future Work
Despite its strengths, ID-Crafter has limitations in modeling highly complex physical interactions and fine-grained dynamics between subjects. It can also inherit biases present in its pretrained components (like the VLM and VAE).
The authors note that future work will focus on incorporating physics-aware priors and advancing fine-grained controllable generation of specific attributes, actions, and complex multi-entity interactions.
Summary updated